abstract type SemObserved

Supertype of all objects that can serve as the observed field of a SEM. Pre-processes data and computes sufficient statistics for example. If you have a special kind of data, you should implement a subtype of SemObserved.

For observed data without missings.

Constructor

SemObservedData(;
    data,
    observed_vars = nothing,
    specification = nothing,
    kwargs...)

Arguments

• data: observed data – DataFrame or Matrix
• observed_vars::Vector{Symbol}: column names of the data (if the object passed as data does not have column names, i.e. is not a data frame)
• specification: optional SEM specification (SemSpecification)

Extended help

Interfaces

• nsamples(::SemObservedData) -> number of observed data points

• nobserved_vars(::SemObservedData) -> number of observed (manifested) variables

• samples(::SemObservedData) -> observed data

• obs_cov(::SemObservedData) -> observed covariance matrix

• obs_mean(::SemObservedData) -> observed mean vector

Type alias for SemObservedData that has mean and covariance, but no actual data.

For instances of SemObservedCovariance samples returns nothing.

SemObservedMissing{T <: Real, S <: Real} <: SemObserved

SemObserved implementation for data with missing values.

Constructor

SemObservedMissing(;
    data,
    observed_vars = nothing,
    specification = nothing,
    lazy_cov = true,
    em_kwargs...)

Arguments

• data: observed data
• observed_vars::Vector{Symbol}: column names of the data (if the object passed as data does not have column names, i.e. is not a data frame)
• specification: optional SEM model specification (SemSpecification)
• lazy_cov::Bool: whether to defer covariance and mean calculation until requested (default: true)
• em_kwargs...: keyword arguments to pass to the EM algorithm (see em_mvn)

SemObservedMissing could be used in combination with SemFIML loss for the full information maximum likelihood (FIML) to fit SEM with missing data. It could also be used with other loss functions, e.g. SemML; in that case the approximated observed covariance and mean would be calculated using the EM algorithm (see em_mvn).

samples(observed::SemObservedData)

Gets the matrix of observed data samples. Rows are samples, columns are observed variables.

See Also

nsamples, observed_vars.

observed_vars(semobj) -> Vector{Symbol}

Return the vector of SEM model observed variable in the order specified by the model, which also should match the order of variables in SemObserved.

abstract type SemSpecification end

Base type for all SEM specifications.

em_mvn(patterns::AbstractVector{SemObservedMissingPattern};
       max_iter_em = 100,
       rtol_em = 1e-4,
       max_nsamples_em = nothing,
       min_eigval = nothing,
       start_em = start_em_observed,
       start_kwargs...)

Estimate the covariance and the mean for data with missing values using the expectation maximization (EM) algorithm.

Arguments

• patterns: the observed data with missing values, grouped by missingness pattern (each pattern is a SemObservedMissingPattern)
• max_iter_em: the maximum number of EM iterations
• rtol_em: the relative tolerance for convergence of the EM algorithm
• max_nsamples_em: the maximum number of samples to use for each pattern in each EM iteration, by default all samples are used, but for large datasets it may be desirable to use a random subset of the data for each pattern in each EM iteration to speed up the algorithm
• min_eigval: the minimum eigenvalue for the covariance matrix; if not nothing, the covariance matrix is regularized in each EM iteration to ensure that all eigenvalues are not smaller than min_eigval, which can help with convergence;
• start_em: the function to generate starting values for the EM algorithm, by default start_em_observed which uses the mean and covariance of the full cases if available
• start_kwargs...: keyword arguments to pass to the start_em function

Returns the tuple of the covariance matrix and the mean vector for the estimated multivariate normal (MVN) distribution.

References

Based on the EM algorithm for MVN-distributed data with missing values adapted from the supplementary material to the book Machine Learning: A Probabilistic Perspective, copyright (2010) Kevin Murphy and Matt Dunham: see gaussMissingFitEm.m and emAlgo.m scripts.

Supertype of all objects that can serve as the implied field of a SEM. Computes model-implied values that should be compared with the observed data to find parameter estimates, e. g. the model implied covariance or mean. If you would like to implement a different notation, e.g. LISREL, you should implement a subtype of SemImplied.

Model implied covariance and means via RAM notation.

Constructor

RAM(specification; gradient = true, kwargs...)

Arguments

• specification: either a RAMMatrices or ParameterTable object
• gradient::Bool: is gradient-based optimization used

Extended help

RAM notation

The model implied covariance matrix is computed as

\[ \Sigma = F(I-A)^{-1}S(I-A)^{-T}F^T\]

and for models with a meanstructure, the model implied means are computed as

\[ \mu = F(I-A)^{-1}M\]

Interfaces

• param_labels(::RAM)-> vector of parameter labels

• nparams(::RAM) -> number of parameters

• ram.Σ -> model implied covariance matrix

• ram.μ -> model implied mean vector

RAM matrices for the current parameter values:

• ram.A
• ram.S
• ram.F
• ram.M

Jacobians of RAM matrices w.r.t to the parameter vector θ

• ram.∇A -> $∂vec(A)/∂θᵀ$
• ram.∇S -> $∂vec(S)/∂θᵀ$
• ram.∇M = $∂M/∂θᵀ$

Vector of indices of each parameter in the respective RAM matrix:

• ram.A_indices
• ram.S_indices
• ram.M_indices

Additional interfaces

• F⨉I_A⁻¹(::RAM) -> $F(I-A)^{-1}$
• F⨉I_A⁻¹S(::RAM) -> $F(I-A)^{-1}S$
• I_A(::RAM) -> $I-A$

Only available in gradient! calls:

• ram.I_A⁻¹ -> $(I-A)^{-1}$

Subtype of SemImplied that implements the RAM notation with symbolic precomputation.

Constructor

RAMSymbolic(
    specification;
    vech = false,
    gradient = true,
    hessian = false,
    approximate_hessian = false,
    kwargs...)

Arguments

• specification: either a RAMMatrices or ParameterTable object
• gradient::Bool: is gradient-based optimization used
• hessian::Bool: is hessian-based optimization used
• approximate_hessian::Bool: for hessian based optimization: should the hessian be approximated
• vech::Bool: should the half-vectorization of Σ be computed (instead of the full matrix) (automatically set to true if any of the loss functions is SemWLS)

Extended help

Interfaces

• param_labels(::RAMSymbolic)-> vector of parameter ids

• nparams(::RAMSymbolic) -> number of parameters

• ram.Σ -> model implied covariance matrix

• ram.μ -> model implied mean vector

Jacobians (only available in gradient! calls)

• ram.∇Σ -> $∂vec(Σ)/∂θᵀ$

• ram.∇μ -> $∂μ/∂θᵀ$

• ∇Σ_eval!(::RAMSymbolic) -> function to evaluate ∇Σ in place, i.e. ∇Σ_eval!(∇Σ, θ). Typically, you do not want to use this but simply query ram.∇Σ.

Hessians The computation of hessians is more involved. Therefore, we desribe it in the online documentation, and the respective interfaces are omitted here.

RAM notation

The model implied covariance matrix is computed as

\[ \Sigma = F(I-A)^{-1}S(I-A)^{-T}F^T\]

and for models with a meanstructure, the model implied means are computed as

\[ \mu = F(I-A)^{-1}M\]

Empty placeholder for models that don't need an implied part. (For example, models that only regularize parameters.)

Constructor

ImpliedEmpty(specification; kwargs...)

Arguments

• specification: either a RAMMatrices or ParameterTable object

Examples

A multigroup model with ridge regularization could be specified as a Sem with one SEM term (SemLoss) per group and an additional SemRidge regularization term.

Extended help

Interfaces

• param_labels(::ImpliedEmpty)-> Vector of parameter labels
• nparams(::ImpliedEmpty) -> Number of parameters

Supertype for all loss functions of SEMs. If you want to implement a custom loss function, it should be a subtype of AbstractLoss.

abstract type SemLoss{O <: SemObserved, I <: SemImplied} <: AbstractLoss

The base type for calculating the loss of the implied SEM model when explaining the observed data.

All subtypes of SemLoss should have the following fields:

• observed::O: object of subtype SemObserved.
• implied::I: object of subtype SemImplied.

Maximum likelihood estimation.

Constructor

SemML(observed, implied, refloss = nothing; approximate_hessian = false)

Arguments

• observed::SemObserved: the observed part of the model
• implied::SemImplied: SemImplied instance
• refloss::Union{SemML, Nothing}: optional reference loss used to preserve loss-specific configuration and share the internal state when rebuilding a loss term, e.g. in replace_observed
• approximate_hessian::Bool: if hessian-based optimization is used, should the hessian be swapped for an approximation

Examples

my_ml = SemML(my_observed, my_implied)

Interfaces

Analytic gradients are available, and for models without a meanstructure and RAMSymbolic implied type, also analytic hessians.

SemFIML{O, I, T, W} <: SemLoss{O, I}

Full information maximum likelihood (FIML) estimation. Can handle observed data with missing values.

Constructor

SemFIML(observed::SemObservedMissing, implied::SemImplied, refloss = nothing)

Arguments

• observed::SemObservedMissing: the observed part of the model (see SemObservedMissing)
• implied::SemImplied: the implied part of the model (see SemImplied)
• refloss::Union{SemFIML, Nothing}: optional reference loss used to preserve loss-specific configuration and share the internal state when rebuilding a loss term, e.g. in replace_observed

Examples

my_fiml = SemFIML(my_observed, my_implied)

Interfaces

Analytic gradients are available.

Weighted least squares estimation. At the moment only available with the RAMSymbolic implied type.

Constructor

SemWLS(
    observed::SemObserved, implied::SemImplied, refloss = nothing;
    wls_weight_matrix = nothing,
    wls_weight_matrix_mean = nothing,
    approximate_hessian = false,
    kwargs...)

Arguments

• observed: the SemObserved part of the model
• implied: the SemImplied part of the model
• refloss::Union{SemWLS, Nothing}: optional reference loss used to preserve loss-specific configuration and share the internal state when rebuilding a loss term, e.g. in replace_observed
• approximate_hessian::Bool: should the hessian be swapped for an approximation
• wls_weight_matrix: the weight matrix for weighted least squares. Defaults to GLS estimation ($0.5*(D^T*kron(S,S)*D)$ where D is the duplication matrix and S is the inverse of the observed covariance matrix)
• wls_weight_matrix_mean: the weight matrix for the mean part of weighted least squares. Defaults to GLS estimation (the inverse of the observed covariance matrix)

Examples

my_wls = SemWLS(my_observed, my_implied)

Interfaces

Analytic gradients are available, and for models without a meanstructure also analytic hessians.

Ridge regularization.

Constructor

SemRidge(;α_ridge, which_ridge, nparams, parameter_type = Float64, implied = nothing, kwargs...)

Arguments

• α_ridge: hyperparameter for penalty term
• which_ridge::Vector: Vector of parameter labels (Symbols) or indices that indicate which parameters should be regularized.
• nparams::Int: number of parameters of the model
• implied::SemImplied: implied part of the model
• parameter_type: type of the parameters

Examples

my_ridge = SemRidge(;α_ridge = 0.02, which_ridge = [:λ₁, :λ₂, :ω₂₃], nparams = 30, implied = my_implied)

Interfaces

Analytic gradients and hessians are available.

SemConstant{C <: Number} <: AbstractLoss

Constant loss term. Can be used for comparability to other packages.

Constructor

SemConstant(;constant_loss, kwargs...)

Arguments

• constant_loss::Number: constant to add to the objective

Examples

    my_constant = SemConstant(42.0)

Interfaces

Analytic gradients and hessians are available.

optimizer_engines()

Returns a vector of optimizer engines supported by the engine keyword argument of the SemOptimizer constructor.

The list of engines depends on the Julia packages loaded (with the using directive) into the current session.

optimizer_engine(::Type{<:SemOptimizer})
optimizer_engine(::SemOptimizer)

Returns the engine name (Symbol) for a SemOptimizer instance or subtype.

optimizer_engine_doc(engine::Symbol)

Shows documentation for the optimizer engine.

For a list of available engines, call optimizer_engines.

SemOptimizer(args...; engine::Symbol = :Optim, kwargs...)

Constructs a SemOptimizer object that can be passed to fit for specifying aspects of the numerical optimization involved in fitting a SEM.

The keyword engine controlls which Julia package is used, with :Optim being the default.

• optimizer_engines() prints a list of currently available engines.
• optimizer_engine_doc(EngineName) prints information on the usage of a specific engine.

More engines become available if specific packages are loaded, for example NLopt.jl (also see Constrained optimization in the online documentation) or ProximalAlgorithms.jl (also see Regularization in the online documentation).

The arguments args... and kwargs... are engine-specific and control further aspects of the optimization process, such as the algorithm, convergence criteria or constraints. Information on those can be accessed with optimizer_engine_doc.

Custom optimizer types shows how to connect the SEM.jl package to a completely new optimization engine.

SemOptimizer(;
    engine = :Optim,
    algorithm = LBFGS(),
    options = Optim.Options(;f_reltol = 1e-10, x_abstol = 1.5e-8),
    kwargs...)

Connects to Optim.jl as the optimization engine.

For more information on the available algorithms and options, see the Optim.jl docs.

Arguments

• algorithm: optimization algorithm from Optim.jl
• options::Optim.Options: options for the optimization algorithm

Examples

# hessian based optimization with backtracking linesearch and modified initial step size
using Optim, LineSearches

my_newton_optimizer = SemOptimizer(
    engine = :Optim,
    algorithm = Newton(
        ;linesearch = BackTracking(order=3),
        alphaguess = InitialHagerZhang()
    )
)

Constrained optimization

When using the Fminbox or SAMIN constrained optimization algorithms, the vector or dictionary of lower and upper bounds for each model parameter can be specified via lower_bounds and upper_bounds keyword arguments. Alternatively, the lower_bound and upper_bound keyword arguments can be used to specify the default bound for all non-variance model parameters, and the variance_lower_bound and variance_upper_bound keyword – for the variance parameters (the diagonal of the S matrix).

SemOptimizer(;
    engine = :NLopt,
    algorithm = :LD_LBFGS,
    options = Dict{Symbol, Any}(),
    local_algorithm = nothing,
    local_options = Dict{Symbol, Any}(),
    equality_constraints = nothing,
    inequality_constraints = nothing,
    constraint_tol::Number = 0.0,
    kwargs...)

Uses NLopt.jl as the optimization engine. For more information on the available algorithms and options, see the NLopt.jl package and the NLopt docs.

Arguments

• algorithm: optimization algorithm.
• options::Dict{Symbol, Any}: options for the optimization algorithm
• local_algorithm: local optimization algorithm
• local_options::Dict{Symbol, Any}: options for the local optimization algorithm
• `equality_constraints: optional equality constraints
• `inequality_constraints:: optional inequality constraints
• constraint_tol::Number: default tolerance for constraints

Constraints specification

Equality and inequality constraints arguments could be a single constraint or any iterable constraints container (e.g. vector or tuple). Each constraint could be a function or any other callable object that takes the two input arguments:

• the vector of the model parameters;
• the array for the in-place calculation of the constraint gradient.

To override the default tolerance, the constraint can be specified as a pair of the function and its tolerance: constraint_func => tol. For information on how to use inequality and equality constraints, see Constrained optimization in our online documentation.

Example

my_optimizer = SemOptimizer(engine = :NLopt)

# constrained optimization with augmented lagrangian
my_constrained_optimizer = SemOptimizer(;
    engine = :NLopt,
    algorithm = :AUGLAG,
    local_algorithm = :LD_LBFGS,
    local_options = Dict(:ftol_rel => 1e-6),
    inequality_constraints = (my_constraint => tol),
)

Interfaces

• algorithm(::SemOptimizerNLopt)
• local_algorithm(::SemOptimizerNLopt)
• options(::SemOptimizerNLopt)
• local_options(::SemOptimizerNLopt)
• equality_constraints(::SemOptimizerNLopt)
• inequality_constraints(::SemOptimizerNLopt)

SemOptimizerProximal(;
    algorithm = ProximalAlgorithms.PANOC(),
    operator_g,
    operator_h = nothing,
    kwargs...,
)

Connects to ProximalAlgorithms.jl as the optimization backend. For more information on the available algorithms and options, see the online docs on Regularization and the documentation of ProximalAlgorithms.jl / ProximalOperators.jl.

Arguments

• algorithm: proximal optimization algorithm.
• operator_g: proximal operator (e.g., regularization penalty)
• operator_h: optional second proximal operator