gluonts.nursery.glide package

gluonts.nursery.glide

Glide is a simple pipeline, which is able to map functions in parallel.

The core idea is to work on already sharded input-data, since inter-process communication is expensive using multiprocessing. Thus, glide’s only syncronisation point is a final result queue to which each process emits its data.

To help splitting the input preemptively glides offers a partition-method.

from gluonts.nursery import glide

data = range(100) parts = glide.partition(data, 3)

def double(n):

return n * 2

assert set(glide.ParApply(glide.lift(double), parts)) == set(range(0, 200, 2))

gluonts.nursery.glide.partition(xs, n)[source]
class gluonts.nursery.glide.Apply(fn, partitions: list, *args, **kwargs)[source]

Bases: object

class gluonts.nursery.glide.ParApply(fn, emitter, batch_size=1, queue_size=50, encode=<function identity>, decode=<function identity>)[source]

Bases: object

gluonts.nursery.glide.lift
class gluonts.nursery.glide.Map(fn, xs)[source]

Bases: object

class gluonts.nursery.glide.Pipeline(fns=())[source]

Bases: object

and_then(*fns)[source]
and_then_each(*fns)[source]
apply(parts, *args, **kwargs)[source]
but_first(*fns)[source]
but_first_each(*fns)[source]
parapply(parts, *args, **kwargs)[source]
gluonts.nursery.glide.imap_unordered(fn, data, num_workers, batch_size=1)[source]

Submodules