gluonts.nursery.glide package

gluonts.nursery.glide

Glide is a simple pipeline, which is able to map functions in parallel.

The core idea is to work on already sharded input-data, since inter-process communication is expensive using multiprocessing. Thus, glide’s only syncronisation point is a final result queue to which each process emits its data.

To help splitting the input preemptively glides offers a partition-method.

from gluonts.nursery import glide

data = range(100) parts = glide.partition(data, 3)

def double(n):

return n * 2

assert set(glide.ParApply(glide.lift(double), parts)) == set(range(0, 200, 2))

class gluonts.nursery.glide.Apply(fn, partitions: list, *args, **kwargs)

Bases: object

class gluonts.nursery.glide.Map(fn, xs)

Bases: object

class gluonts.nursery.glide.ParApply(fn, emitter, batch_size=1, queue_size=50, encode=<function identity>, decode=<function identity>)

Bases: object

class gluonts.nursery.glide.Pipeline(fns=())

Bases: object

and_then(*fns)

and_then_each(*fns)

apply(parts, *args, **kwargs)

but_first(*fns)

but_first_each(*fns)

parapply(parts, *args, **kwargs)

gluonts.nursery.glide.divide_into(length, n)

gluonts.nursery.glide.imap_unordered(fn, data, num_workers, batch_size=1)

gluonts.nursery.glide.lift(fn='__no__default__', xs='__no__default__')

gluonts.nursery.glide.partition(xs, n)

Submodules

• gluonts.nursery.glide.parallel module
• gluonts.nursery.glide.pipeline module
• gluonts.nursery.glide.sequential module
• gluonts.nursery.glide.util module