| Title: | Measuring Contributions of Component Models to Ensemble Forecast Accuracy |
|---|---|
| Description: | Provides metrics for quantifying the contribution of individual component models to the predictive accuracy of ensemble forecasts. |
| Authors: | Minsu Kim [aut, cre] (ORCID: <https://orcid.org/0009-0008-4637-3589>), Li Shandross [aut, ctb] (ORCID: <https://orcid.org/0009-0008-1348-1954>), Zhian Kamvar [ctb] (ORCID: <https://orcid.org/0000-0003-1458-7108>), Nicholas Reich [aut] (ORCID: <https://orcid.org/0000-0003-3503-9899>), Evan Ray [aut] |
| Maintainer: | Minsu Kim <[email protected]> |
| License: | MIT + file LICENSE |
| Version: | 0.1.0 |
| Built: | 2026-07-17 06:32:10 UTC |
| Source: | https://github.com/mkim425/modelimportance |
Aggregate model importance scores across tasks to compute overall importance for each model
## S3 method for class 'model_imp_tbl' aggregate( x, by = "model_id", na_action = c("drop", "worst", "average"), fun = mean, ... )## S3 method for class 'model_imp_tbl' aggregate( x, by = "model_id", na_action = c("drop", "worst", "average"), fun = mean, ... )
x |
An object of class |
by |
A character vector with column names specifying the grouping
variable(s) for summarization. Default is |
na_action |
A character string specifying how to handle
|
fun |
A function used to summarize importance scores.
Default is |
... |
Additional arguments passed to the summary function |
This method extends stats::aggregate for objects of class model_imp_tbl.
A data frame with columns model_id and importance_score_<fun>,
where <fun> is the name of the summary function
used (e.g., importance_score_mean when fun = mean).
The output is sorted in descending order of the summary importance scores.
A subset of hubExamples::forecast_outputs filtered to quantile output type
and location "25" (Massachusetts), covering the 2022/23 influenza season
(from "2022-11-19" to "2023-01-07").
forecast_data_exampleforecast_data_example
A data frame (model_out_tbl) with 168 rows and 9 columns:
Character. Unique model identifier.
Date. Date the forecast was generated.
Character. Forecast target.
Integer. Number of weeks ahead.
Character. FIPS code for the US location.
Date. Target date for the forecast.
Character. Prediction representation type.
Character. Identifier for the output type level.
Numeric. Predicted value.
Sourced from hubExamples::forecast_outputs
(hubverse-org/hubExamples v1.0.0).
A subset of hubExamples::forecast_outputs filtered to median output type,
location "25" (Massachusetts), horizon 1, and target end date 2022-12-24.
forecast_data_ma_h1forecast_data_ma_h1
A data frame (model_out_tbl) with 3 rows and 9 columns:
Character. Unique model identifier.
Date. Date the forecast was generated.
Character. Forecast target.
Integer. Number of weeks ahead.
Character. FIPS code for the US location.
Date. Target date for the forecast.
Character. Prediction representation type.
Character. Identifier for the output type level.
Numeric. Predicted value.
Sourced from hubExamples::forecast_outputs
(hubverse-org/hubExamples v1.0.0).
A subset of hubExamples::forecast_outputs filtered to median output type
and two target end dates (2022-11-26, 2022-12-10).
forecast_data_rawforecast_data_raw
A data frame (model_out_tbl) with 12 rows and 9 columns:
Character. Unique model identifier.
Date. Date the forecast was generated.
Character. Forecast target.
Integer. Number of weeks ahead.
Character. FIPS code for the US location.
Date. Target date for the forecast.
Character. Prediction representation type.
Character. Identifier for the output type level.
Numeric. Predicted value.
Sourced from hubExamples::forecast_outputs
(hubverse-org/hubExamples v1.0.0).
We measure each ensemble component model's contribution to the ensemble prediction accuracy for each model task.
This function requires that one column represent the forecast date (or
a date from which each forecast originates or is made in reference to) and
that column be named one of forecast_date, origin_date, and
reference_date.
For each output_type, the corresponding scoring rule applied to calculate
the importance is as follows.
| Output Type | Scoring Rule | Description |
| mean | se_point | evaluate using the squared error |
| median | ae_point | evaluate using the absolute error |
| quantile | wis | evaluate using the weighted interval score |
| pmf | log_score | evaluate using the logarithm of the probability assigned to the true outcome |
model_importance( forecast_data, oracle_output_data, ensemble_fun = c("simple_ensemble", "linear_pool"), importance_algorithm = c("lomo", "lasomo"), subset_wt = c("equal", "perm_based"), min_log_score = -10, ... )model_importance( forecast_data, oracle_output_data, ensemble_fun = c("simple_ensemble", "linear_pool"), importance_algorithm = c("lomo", "lasomo"), subset_wt = c("equal", "perm_based"), min_log_score = -10, ... )
forecast_data |
A data.frame with the predictions that is or can be
coerced to a |
oracle_output_data |
Ground truth data for the variables that are used to define modeling targets. This data must follow the oracle output format. See 'Details'. |
ensemble_fun |
A character string specifying a ensemble method, either
"simple_ensemble" or "linear_pool";
|
importance_algorithm |
A character string specifying algorithm for model
importance calculation; |
subset_wt |
A character string specifying method for assigning weight
to subsets when using
|
min_log_score |
A numeric value specifying a minimum threshold for log
scores for the |
... |
Optional arguments passed to |
The oracle_output_data is a data frame that contains the ground truth
values for the variables used to define modeling targets. It is referred to
as “oracle” because it is formatted as if an oracle made a perfect point
prediction equal to the truth. This data must follow the oracle output format
defined in the hubverse standard, which includes independent task ID columns
(e.g., location, target_date), the output_type column specifying the
output type of the predictions and an oracle_value column for the observed
values. As in the forecast data, if the output_type is either "quantile"
or "pmf", the output_type_id column is often required to provide further
identifying information.
The model_out_tbl and oracle_output_data must have the same task ID
columns and output_type, including output_type_id if necessary, which are
used to match the predictions with the ground truth data.
Additional argument in ... is agg_fun, which is a character string name
for a function specifying aggregation method of component model outputs.
Default is mean, indicating that equally weighted mean is calculated
across all component model outputs for each unique output_type_id.
This can be median or a custom function (e.g., geometric_mean. Details
can be found in
https://hubverse-org.github.io/hubEnsembles/articles/hubEnsembles.html).
This function uses the furrr and future for parallelization.
To enable parallel execution, please set a parallel backend, e.g., via
future::plan().
A model_imp_tbl S3 class object with columns model_id,
reference_date, output_type, and importance, along with any task ID
columns (e.g., location, horizon, and target_end_date) present in the
input forecast_data.
Note that reference_date is used as the name for the forecast date column,
regardless of its original name in the input forecast_data.
Optional progress bars are displayed via the progressr package when it is
installed and the session is interactive.
If progressr is not installed, the function will run without progress bars.
To enable progress bars,
progressr::handlers(global = TRUE)
model importance scores calculated for individual prediction tasks can be
aggregated across multiple tasks to obtain an overall importance score for
each model via the aggregate() method for model_imp_tbl objects.
Users can summarize the task-level scores with specified summary statistics
of interest (e.g., mean, median, quantiles) for a more comprehensive view of
model importance.
See aggregate.model_imp_tbl() for more details on how to use this method.
## Not run: library(dplyr) forecast_data <- forecast_data_raw |> dplyr::filter( location == "25", horizon == 1 ) target_data <- target_data_raw |> dplyr::filter( target_end_date %in% unique(forecast_data$target_end_date), location == "25" ) # Example with the default arguments. model_importance( forecast_data = forecast_data, oracle_output_data = target_data, ensemble_fun = "simple_ensemble", importance_algorithm = "lomo", subset_wt = "equal" ) # Example with the additional argument in `...`. model_importance( forecast_data = forecast_data, oracle_output_data = target_data, ensemble_fun = "simple_ensemble", importance_algorithm = "lomo", subset_wt = "equal", agg_fun = median ) ## End(Not run)## Not run: library(dplyr) forecast_data <- forecast_data_raw |> dplyr::filter( location == "25", horizon == 1 ) target_data <- target_data_raw |> dplyr::filter( target_end_date %in% unique(forecast_data$target_end_date), location == "25" ) # Example with the default arguments. model_importance( forecast_data = forecast_data, oracle_output_data = target_data, ensemble_fun = "simple_ensemble", importance_algorithm = "lomo", subset_wt = "equal" ) # Example with the additional argument in `...`. model_importance( forecast_data = forecast_data, oracle_output_data = target_data, ensemble_fun = "simple_ensemble", importance_algorithm = "lomo", subset_wt = "equal", agg_fun = median ) ## End(Not run)
Print method for model importance score table
## S3 method for class 'model_imp_tbl' print(x, ...)## S3 method for class 'model_imp_tbl' print(x, ...)
x |
An object of class |
... |
Additional arguments passed to the print method. |
Print method for summary of model importance score table
## S3 method for class 'summary.model_imp_tbl' print(x, ...)## S3 method for class 'summary.model_imp_tbl' print(x, ...)
x |
An object of class |
... |
Additional arguments passed to the print method. |
Summary method for model importance score table
## S3 method for class 'model_imp_tbl' summary(object, ...)## S3 method for class 'model_imp_tbl' summary(object, ...)
object |
An object of class |
... |
Additional arguments passed to the print method. |
A subset of hubExamples::forecast_target_ts matching the locations and
target end dates in forecast_data_example.
target_data_exampletarget_data_example
A data frame with 8 rows and 4 columns:
Date. Date of the observation.
Character. Target name.
Character. FIPS code for the US location.
Numeric. Observed value.
Sourced from hubExamples::forecast_target_ts
(hubverse-org/hubExamples v1.0.0).
A subset of hubExamples::forecast_target_ts for location "25"
(Massachusetts) and target end date 2022-12-24.
target_data_matarget_data_ma
A data frame with 1 row and 4 columns:
Date. Date of the observation.
Character. Target name.
Character. FIPS code for the US location.
Numeric. Observed value.
Sourced from hubExamples::forecast_target_ts
(hubverse-org/hubExamples v1.0.0).
A subset of hubExamples::forecast_target_ts matching the locations and
target end dates in forecast_data_raw, with the observed values column
renamed to oracle_value.
target_data_rawtarget_data_raw
A data frame with 4 rows and 4 columns:
Date. Date of the observation.
Character. Target name.
Character. FIPS code for the US location.
Numeric. Observed value.
Sourced from hubExamples::forecast_target_ts
(hubverse-org/hubExamples v1.0.0).