gluonts.dataset.loader module

class gluonts.dataset.loader.DataLoader(dataset: Iterable[Dict[str, Any]], *, transform: gluonts.transform._base.Transformation, cyclic: bool, is_train: bool, batch_size: int, ctx: mxnet.context.Context, dtype: gluonts.core.component.DType = <class 'numpy.float32'>, num_workers: Optional[int] = None, num_prefetch: Optional[int] = None, **kwargs)[source]

Bases: typing.Iterable

An abstract Iterable type for iterating and transforming a dataset, in batches of a prescribed size.

Parameters
  • dataset – The dataset from which to load data.

  • transform – A transformation to apply to each entry in the dataset.

  • batch_size – The size of the batches to emit.

  • ctx – MXNet context to use to store data.

  • dtype – Floating point type to use.

  • num_workers – The number of multiprocessing workers to use for data preprocessing. By default 0, in which case no multiprocessing will be utilized.

  • num_prefetch – The number of prefetching batches only works if num_workers > 0. If prefetch > 0, it allow worker process to prefetch certain batches before acquiring data from iterators. Note that using large prefetching batch will provide smoother bootstrapping performance, but will consume more shared_memory. Using smaller number may forfeit the purpose of using multiple worker processes, try reduce num_workers in this case. By default it defaults to num_workers * 2.

  • cyclic – Indicates whether the dataset is traversed potentially multiple times.

class gluonts.dataset.loader.InferenceDataLoader(dataset: Iterable[Dict[str, Any]], *, transform: gluonts.transform._base.Transformation, batch_size: int, ctx: mxnet.context.Context, num_workers: Optional[int] = None, num_prefetch: Optional[int] = None, dtype: gluonts.core.component.DType = <class 'numpy.float32'>, **kwargs)[source]

Bases: gluonts.dataset.loader.DataLoader

class gluonts.dataset.loader.TrainDataLoader(dataset: Iterable[Dict[str, Any]], transform: gluonts.transform._base.Transformation, batch_size: int, ctx: mxnet.context.Context, num_batches_per_epoch: int, num_workers: Optional[int] = None, num_prefetch: Optional[int] = None, dtype: gluonts.core.component.DType = <class 'numpy.float32'>, **kwargs)[source]

Bases: gluonts.dataset.loader.DataLoader

An Iterable type for iterating and transforming a dataset, in batches of a prescribed size, until a given number of batches is reached.

The transformation are applied with in training mode, i.e. with the flag is_train = True.

Parameters
  • dataset – The dataset from which to load data.

  • transform – A transformation to apply to each entry in the dataset.

  • batch_size – The size of the batches to emit.

  • ctx – MXNet context to use to store data.

  • num_batches_per_epoch – Number of batches to return in one complete iteration over this object.

  • num_workers – The number of multiprocessing workers to use for data preprocessing. By default 0, in which case no multiprocessing will be utilized.

  • num_prefetch – The number of prefetching batches only works if num_workers > 0. If prefetch > 0, it allow worker process to prefetch certain batches before acquiring data from iterators. Note that using large prefetching batch will provide smoother bootstrapping performance, but will consume more shared_memory. Using smaller number may forfeit the purpose of using multiple worker processes, try reduce num_workers in this case. By default it defaults to num_workers * 2.

  • dtype – Floating point type to use. Default is np.float32.

class gluonts.dataset.loader.ValidationDataLoader(dataset: Iterable[Dict[str, Any]], *, transform: gluonts.transform._base.Transformation, batch_size: int, ctx: mxnet.context.Context, num_workers: Optional[int] = None, num_prefetch: Optional[int] = None, dtype: gluonts.core.component.DType = <class 'numpy.float32'>, **kwargs)[source]

Bases: gluonts.dataset.loader.DataLoader