Post-training assessment

The function assess can be used to assess a trained estimator. The resulting Assessment object contains ground-truth parameters, estimates, and other quantities that can be used to compute quantitative and qualitative diagnostics.

NeuralEstimators.assess Function

assess(estimator, θ, Z; ...)
assess(estimators::Vector, θ, Z; ...)

Assesses an estimator (or a collection of estimators) based on true parameters θ and corresponding simulated data Z.

The parameters θ should be given as a × matrix, where is the parameter dimension and is the number of sampled parameter vectors.

When Z contains more simulated data sets than the number of sampled parameter vectors, θ will be recycled via horizontal concatenation: θ = repeat(θ, outer = (1, J)), where J = numobs(Z) ÷ K is the number of simulated data sets for each parameter vector. This allows assessment of the estimator's sampling distribution under fixed parameters.

The return value is of type Assessment.

Keyword arguments

• estimator_name::String (or estimator_names::Vector{String} for multiple estimators): name(s) of the estimator(s) (sensible defaults provided).

• parameter_names::Vector{String}: names of the parameters (sensible default provided).

• use_gpu = true: Bool or Vector{Bool} with length equal to the number of estimators.

• probs = nothing (applicable only to PointEstimator): probability levels taking values between 0 and 1. By default, no bootstrap uncertainty quantification is done; if probs is provided, it must be a two-element vector specifying the lower and upper probability levels for non-parametric bootstrap intervals (note that parametric bootstrap is not currently supported with assess()).

• B::Integer = 400 (applicable only to PointEstimator): number of bootstrap samples.

• pointsummary::Function = mean (applicable only to estimators that yield posterior samples): a function that summarises a vector of posterior samples into a single point estimate for each marginal; any function mapping a vector to a scalar is valid (e.g., median for the posterior median).

• kwargs... (applicable only to estimators that yield posterior samples): additional keyword arguments passed to sampleposterior.

source

NeuralEstimators.Assessment Type

Assessment

A type for storing the output of assess(). The field runtime contains the total time taken for each estimator. The field estimates is a long-form DataFrame with columns:

• parameter: the name of the parameter

• truth: the true value of the parameter

• estimate: the estimated value of the parameter

• k: the index of the parameter vector

• j: the index of the data set (only relevant in the case that multiple data sets are associated with each parameter vector)

If the estimator is a PosteriorEstimator or a RatioEstimator, in addition to the fields listed above, the field samples stores the posterior samples as a long-form DataFrame with the columns parameter, truth, k, j (as given above), as well as:

• draw: the index of the draw within the posterior samples

• value: the value of the posterior sample for a given parameter and draw.

If the estimator is an IntervalEstimator, the column estimate will be replaced by the columns lower and upper, containing the lower and upper bounds of the interval, respectively.

If the estimator is a QuantileEstimator, there will also be a column prob indicating the probability level of the corresponding quantile estimate.

Use merge() to combine assessments from multiple estimators of the same type or join() to combine assessments from a PointEstimator and an IntervalEstimator.

source

Makie.plot Method

plot(assessment::Assessment; plots = nothing, ...)

Visualise the performance of a neural estimator, given the Assessment object returned by assess. Returns a Makie Figure with one panel for each parameter.

Extension

This function is defined in the NeuralEstimatorsPlottingMakieExt extension and requires CairoMakie (or another Makie backend) to be loaded.

The plots that are available depend on the type of estimator that was assessed. By default all of them are drawn; use the keyword argument plots to select a subset.

Point and interval estimates (PointEstimator, IntervalEstimator):

• :recovery: estimates against the true values, with intervals (when the assessment contains them) drawn as vertical line segments. Accurate estimates lie close to the dashed identity line.

Quantile estimates (QuantileEstimator):

• :calibration: the proportion of estimated quantiles that are greater than the true value, against the nominal probability level τ. Specifically, for k = 1,…,K, sample pairs (θᵏ, Zᵏ) with θᵏ ∼ p(θ) and Zᵏ ∼ p(Z ∣ θᵏ), so that θᵏ is a draw from the posterior p(θ ∣ Zᵏ); then, for each τ, plot the proportion of the estimated quantiles Q(Zᵏ, τ) that are greater than θᵏ. A well-calibrated estimator follows the dashed identity line.

Posterior samples (PosteriorEstimator, RatioEstimator, TelescopingRatioEstimator):

• :recovery: point estimates (the pointsummary given to assess, by default the posterior mean) against the true values, with central 95% credible intervals drawn as vertical line segments.

• :ecdf: simulation-based calibration. For each parameter, the empirical distribution function of the fractional rank of the true value among the posterior draws, together with a simultaneous prob-level confidence band (Säilynoja et al., 2022). A well-calibrated posterior gives a curve that stays within the band.

• :zscore: the posterior z-score, (posterior mean − true value) / posterior standard deviation, against the posterior contraction, 1 − posterior variance / prior variance. Ideally the z-scores are centred on zero and the contractions are close to one.

Keyword arguments

• plots = nothing: the plots to draw, given as a Symbol or a collection of Symbols from those listed above, in the order in which they should appear (e.g., plots = (:recovery, :ecdf)). By default, all available plots are drawn.

• prob = 0.99: simultaneous coverage of the confidence band in the :ecdf plot.

• difference = true: if true, the :ecdf plot shows the difference between the empirical distribution function and that of the uniform distribution, which makes departures from calibration easier to see; if false, it shows the empirical distribution function itself.

• grid = false: when the assessment contains several estimators (see merge()), they are by default drawn in the same panels in different colours. If grid = true, each estimator is instead given its own row of panels, which is easier to read with more than three estimators.

• ncols = nothing: the number of panels in each row, after which the parameters wrap onto a new row. By default, at most four, balanced across rows.

• figure = (;), axis = (;): attributes passed to the Figure and to every Axis, respectively. By default the panels are of a fixed size and the figure is sized to fit them; give figure = (; size = (w, h)) to fix the size of the figure instead. Colours and fonts are taken from the current Makie theme.

Examples

using NeuralEstimators, CairoMakie

# Given an estimator and test parameters and data (see `assess`)
assessment = assess(estimator, θ_test, Z_test)

plot(assessment)                               # all available plots
plot(assessment; plots = :recovery)            # a single plot
plot(assessment; plots = (:recovery, :ecdf))   # a subset (posterior samples)

source

NeuralEstimators.risk Function

risk(assessment::Assessment; ...)

Computes a Monte Carlo approximation of an estimator's Bayes risk,

where    denotes a set of parameter vectors sampled from the prior and, for each , data are simulated from the statistical model conditional on .

If the Assessment object corresponds to an estimator with a self-defined loss (e.g., PosteriorEstimator), the precomputed risk is returned directly. Otherwise, the risk is computed from the estimates and true parameters using the provided loss function.

Keyword arguments

• loss = (x, y) -> abs(x - y): a binary operator defining the loss function (default: absolute-error loss)

• average_over_parameters::Bool = false: if true, the loss is averaged over all parameters; otherwise (default), it is computed separately for each parameter.

source

NeuralEstimators.bias Function

bias(assessment::Assessment; average_over_parameters = false)

Computes a Monte Carlo approximation of an estimator's bias,

where    denotes a set of parameter vectors sampled from the prior and, for each , data are simulated from the statistical model conditional on .

source

NeuralEstimators.rmse Function

rmse(assessment::Assessment; average_over_parameters = false)

Computes a Monte Carlo approximation of an estimator's root-mean-squared error,

where    denotes a set of parameter vectors sampled from the prior and, for each , data are simulated from the statistical model conditional on .

source

NeuralEstimators.coverage Function

coverage(assessment::Assessment; ...)

Computes a Monte Carlo approximation of an interval estimator's expected coverage, as defined in Hermans et al. (2022, Definition 2.1), and the proportion of parameters below and above the lower and upper bounds, respectively.

Keyword arguments

• average_over_parameters::Bool = false: if true, the coverage is averaged over all parameters; otherwise (default), it is computed over each parameter separately.

source