gluonts.evaluation package

gluonts.evaluation.get_seasonality[source]

Returns the default seasonality for a given freq str. E.g. for

2H -> 12

class gluonts.evaluation.Evaluator(quantiles: Iterable[Union[float, str]] = (0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9), seasonality: Optional[int] = None, alpha: float = 0.05, calculate_owa: bool = False, num_workers: Optional[int] = None, chunk_size: Optional[int] = None)[source]

Bases: object

Evaluator class, to compute accuracy metrics by comparing observations to forecasts.

Parameters
  • quantiles – list of strings of the form ‘p10’ or floats in [0, 1] with the quantile levels

  • seasonality – seasonality to use for seasonal_error, if nothing is passed uses the default seasonality for the given series frequency as returned by get_seasonality

  • alpha – parameter of the MSIS metric from M4 competition that defines the confidence interval for alpha=0.05 the 95% considered is considered in the metric, see https://www.m4.unic.ac.cy/wp-content/uploads/2018/03/M4 -Competitors-Guide.pdf for more detail on MSIS

  • calculate_owa – Determines whether the OWA metric should also be calculated, which is computationally expensive to evaluate and thus slows down the evaluation process considerably. By default False.

  • num_workers – The number of multiprocessing workers that will be used to process the data in parallel. Default is multiprocessing.cpu_count(). Setting it to 0 means no multiprocessing.

  • chunk_size – Controls the approximate chunk size each workers handles at a time. Default is 32.

static abs_error(target, forecast)[source]
static abs_target_mean(target)[source]
static abs_target_sum(target)[source]
static coverage(target, quantile_forecast)[source]
default_quantiles = (0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9)
static extract_past_data(time_series: Union[pandas.core.series.Series, pandas.core.frame.DataFrame], forecast: gluonts.model.forecast.Forecast) → numpy.ndarray[source]
Parameters
  • time_series

  • forecast

Returns

time series without the forecast dates

Return type

np.ndarray

static extract_pred_target(time_series: Union[pandas.core.series.Series, pandas.core.frame.DataFrame], forecast: gluonts.model.forecast.Forecast) → numpy.ndarray[source]
Parameters
  • time_series

  • forecast

Returns

time series cut in the Forecast object dates

Return type

np.ndarray

get_aggregate_metrics(metric_per_ts: pandas.core.frame.DataFrame) → Tuple[Dict[str, float], pandas.core.frame.DataFrame][source]
get_metrics_per_ts(time_series: Union[pandas.core.series.Series, pandas.core.frame.DataFrame], forecast: gluonts.model.forecast.Forecast) → Dict[str, Union[float, str, None]][source]
static mape(target, forecast)[source]
\[mape = mean(|Y - Y_hat| / |Y|))\]
static mase(target, forecast, seasonal_error)[source]
\[mase = mean(|Y - Y_hat|) / seasonal_error\]

https://www.m4.unic.ac.cy/wp-content/uploads/2018/03/M4-Competitors-Guide.pdf

static mse(target, forecast)[source]
static msis(target, lower_quantile, upper_quantile, seasonal_error, alpha)[source]
Math

msis = mean(U - L + 2/alpha * (L-Y) * I[Y<L] + 2/alpha * (Y-U) * I[Y>U]) /seasonal_error

https://www.m4.unic.ac.cy/wp-content/uploads/2018/03/M4-Competitors-Guide.pdf

static owa(target: numpy.ndarray, forecast: numpy.ndarray, past_data: numpy.ndarray, seasonal_error: float, start_date: pandas._libs.tslibs.timestamps.Timestamp) → float[source]
\[owa = 0.5*(smape/smape_naive + mase/mase_naive)\]

https://www.m4.unic.ac.cy/wp-content/uploads/2018/03/M4-Competitors-Guide.pdf

static quantile_loss(target, quantile_forecast, q)[source]
seasonal_error(past_data: numpy.ndarray, forecast: gluonts.model.forecast.Forecast) → float[source]
\[seasonal_error = mean(|Y[t] - Y[t-m]|)\]

where m is the seasonal frequency https://www.m4.unic.ac.cy/wp-content/uploads/2018/03/M4-Competitors-Guide.pdf

static smape(target, forecast)[source]
\[smape = mean(2 * |Y - Y_hat| / (|Y| + |Y_hat|))\]

https://www.m4.unic.ac.cy/wp-content/uploads/2018/03/M4-Competitors-Guide.pdf

class gluonts.evaluation.MultivariateEvaluator(quantiles: Iterable[Union[float, str]] = array([0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9]), seasonality: Optional[int] = None, alpha: float = 0.05, eval_dims: List[int] = None, target_agg_funcs: Dict[str, Callable] = {})[source]

Bases: gluonts.evaluation._base.Evaluator

The MultivariateEvaluator class owns functionality for evaluating multidimensional target arrays of shape (target_dimensionality, prediction_length).

Evaluations of individual dimensions will be stored with the corresponding dimension prefix and contain the metrics calculated by only this dimension. Metrics with the plain metric name correspond to metrics calculated over all dimensions. Additionally, the user can provide additional aggregation functions that first aggregate the target and forecast over dimensions and then calculate the metric. These metrics will be prefixed with m_<aggregation_fun_name>_

The evaluation dimensions can be set by the user.

Example

{‘0_MSE’: 0.004307240342677687, # MSE of dimension 0 ‘0_abs_error’: 1.6246897801756859, ‘1_MSE’: 0.003949341769475723, # MSE of dimension 1 ‘1_abs_error’: 1.5052175521850586, ‘MSE’: 0.004128291056076705, # MSE of all dimensions ‘abs_error’: 3.1299073323607445, ‘m_sum_MSE’: 0.02 # MSE of aggregated target and aggregated forecast (if target_agg_funcs is set). ‘m_sum_abs_error’: 4.2}

calculate_aggregate_multivariate_metrics(ts_iterator: Iterator[pandas.core.frame.DataFrame], forecast_iterator: Iterator[gluonts.model.forecast.Forecast], agg_fun: Callable) → Dict[str, float][source]
Parameters
  • ts_iterator – Iterator over time series

  • forecast_iterator – Iterator over forecasts

  • agg_fun – aggregation function

Returns

dictionary with aggregate datasets metrics

Return type

Dict[str, float]

calculate_aggregate_vector_metrics(all_agg_metrics: Dict[str, float], all_metrics_per_ts: pandas.core.frame.DataFrame) → Dict[str, float][source]
Parameters
  • all_agg_metrics – dictionary with aggregate metrics of individual dimensions

  • all_metrics_per_ts – DataFrame containing metrics for all time series of all evaluated dimensions

Returns

dictionary with aggregate metrics (of individual (evaluated) dimensions and the entire vector)

Return type

Dict[str, float]

static extract_aggregate_forecast(forecast_iterator: Iterator[gluonts.model.forecast.Forecast], agg_fun: Callable) → Iterator[gluonts.model.forecast.Forecast][source]
static extract_aggregate_target(it_iterator: Iterator[pandas.core.frame.DataFrame], agg_fun: Callable) → Iterator[pandas.core.frame.DataFrame][source]
static extract_forecast_by_dim(forecast_iterator: Iterator[gluonts.model.forecast.Forecast], dim: int) → Iterator[gluonts.model.forecast.Forecast][source]
static extract_target_by_dim(it_iterator: Iterator[pandas.core.frame.DataFrame], dim: int) → Iterator[pandas.core.frame.DataFrame][source]
get_eval_dims(target_dimensionality: int) → List[int][source]
static get_target_dimensionality(forecast: gluonts.model.forecast.Forecast) → int[source]
static peek(iterator: Iterator[Any]) → Tuple[Any, Iterator[Any]][source]