Block Library

class reip.blocks.misc.Iterator(iterator, **kw)

Yield each element in an iterator one at a time.

process(meta=None)

Process data.

class reip.blocks.misc.Interval(seconds=2, initial=None, **kw)

Emit every X seconds.

init()

Initialize the block.

process(meta=None)

Process data.

class reip.blocks.misc.Time(**kw)

Add system time to metadata.

process(x, meta)

Process data.

class reip.blocks.misc.Meta(meta, **kw)

Add arbitrary data to metadata.

process(x, meta=None)

Process data.

class reip.blocks.misc.Glob(*paths, recursive=True, atonce=False, **kw)

Outputs files matching glob patterns.

init()

Initialize the block.

process(meta=None)

Process data.

class reip.blocks.misc.AsDict(*columns, prepare=<function AsDict.<lambda>>, meta=None, **kw)

Merge block outputs into a dict.

process(*xs, meta=None)

Process data.

class reip.blocks.misc.Constant(value, *a, **kw)

Yields a constant value.

process(meta)

Process data.

class reip.blocks.misc.Increment(start=None, stop=None, step=1, **kw)

Yield an incrementing (or decrementing) value.

class reip.blocks.misc.Debug(message=None, level=None, convert=None, value=False, compact=False, summary=False, period=None, name=None, border=False, log_level=None, **kw)

Debug the output of a block.

process(*xs, meta=None)

Process data.

class reip.blocks.misc.Results(squeeze=True, **kw)

Gather the results in lists as self.results and self.meta

init()

Initialize the block.

process(*xs, meta=None)

Process data.

class reip.blocks.misc.Lambda(func=None, name=None, init=None, finish=None, n_inputs=None, **kw)

An inline way to create a block from a regular function. Input and output formats are the same.

init()

Initialize the block.

process(*xs, meta=None)

Process data.

finish()

Cleanup.

class reip.blocks.misc.Interleave(sort_key=None, **kw)

Interleave the block inputs to a single output.

process(*xs, meta=None)

Process data.

class reip.blocks.misc.Separate(bins, **kw)

Separate out a single input into multiple outputs based on a condition per output.

process(x, meta=None)

Process data.

class reip.blocks.misc.Gather(size=None, reduce=None, squeeze=True, **kw)

init()

Initialize the block.

process(*xs, meta=None)

Process data.

finish()

Cleanup.

Buffering

class reip.blocks.buffer.Rebuffer(size=None, duration=None, *a, sr_key='sr', **kw)

Collect numpy arrays and convert them to a different resolution

init()

Initialize the block.

process(x, meta)

Process data.

class reip.blocks.buffer.FastRebuffer(size=None, **kw)

init()

Initialize the block.

process(x, meta)

Process data.

class reip.blocks.buffer.GatedRebuffer(sampler, size=10, **kw)

Rebuffer while also sampling sparsely in time. For example, sampling audio files while leaving gaps between them for privacy reasons.

init()

Initialize the block.

finish()

Cleanup.

process(x, meta)

Process data.

reip.blocks.buffer.temporal_coverage(clip_duration=10, coverage=0.5, min_silence=5.0, sampling='normal')

Given a clip_duration and desired coverage fraction, compute how long the silence between clips needs to be to satisfy the desired coverage and add some random variance to the silence based so that it gives values sampled from the distribution specified by sampling.

Parameters

• clip_duration (float) – Duration of non-silent clip in seconds

• coverage (float) – Fraction of time to be recorded (non-silence), must be in range (0,1].

• min_silence (float) – Minimum silence allowed between clips, in seconds

• sampling (str) – The distribution from which silence durations will be sampled, must be ‘uniform’ or ‘normal’.

Returns

The amount of silence to insert between the current

and next clip, in seconds.

Return type

silence (float)

Shell

class reip.blocks.shell.Shell(cmd, astype=<class 'str'>, **kw)

Run a shell command on the input data. See reip.util.shell.run() for information about command formatting.

process(*xs, meta)

Process data.

class reip.blocks.shell.ShellProcess(cmd, **kw)

Spawn a shell process while this block is running. There is currently no support for passing data to and from the process, but their outputs are available via self.stdout and self.stderr.

init()

Initialize the block.

finish()

Cleanup.

File Watching

File System Events

Watchdog API Reference: https://pythonhosted.org/watchdog/api.html

Note

from experience, I’ve found file watchers can be a bit unreliable - (potentially due to my own misuse !).

Additionally, many times (for example file upload) watching file events may not actually be what you want, and you may be better off polling file lists using glob as that will reduce the number of missed files and is resilient to reboots.

class reip.blocks.os_watch.Watch(*patterns, path='./', event_types=None, recursive=False, **kw)

A Generic File Watcher

init()

Initialize the block.

process(meta)

Process data.

finish()

Cleanup.

class reip.blocks.os_watch.Created(*patterns, path='./', event_types=None, recursive=False, **kw)

Outputs any created files.

class reip.blocks.os_watch.Modified(*patterns, path='./', event_types=None, recursive=False, **kw)

Outputs any modified files.

class reip.blocks.os_watch.Deleted(*patterns, path='./', event_types=None, recursive=False, **kw)

Outputs any deleted files.

class reip.blocks.os_watch.Moved(*patterns, path='./', event_types=None, recursive=False, **kw)

Outputs any moved files.

File Postprocessing

class reip.blocks.archive.Tar(filename='{time}.tar', remove_files=False, gz=None, **kw)

Archive files in a .tar file.

process(*files, meta)

Process data.

class reip.blocks.archive.TarGz(filename='{time}.tgz', **kw)

class reip.blocks.encrypt.TwoStageEncrypt(filename, rsa_key, remove_files=False, extra_files=None, is_rsa_key_file=True, **kw)

Encrypt a file with a fresh generated key, then encrypt the key with the root key. Then tar the encrypted file and the encrypted key together.

process(file, meta)

Encrypt file with AES 4096. :param file: name of the file to be encrypted :param path: Location to save encrypted file :param default: current dir

class reip.blocks.encrypt.TwoStageDecrypt(filename, rsa_key, remove_files=False, **kw)

Decrypt a file encrypted with TwoStepEncryptFile.

process(file, meta=None)

Process data.

decrypt(msg, enc_key)

Decrypt file with AES 4096.

Audio

Video

class reip.blocks.video.core.Video(index=None, file=None, fps=30, size=None, **kw)

init()

Initialize the block.

process(x=None, meta=None)

Process data.

finish()

Cleanup.

class reip.blocks.video.core.VideoWriter(filename, **kw)

init()

Initialize the block.

process(X, meta)

Process data.

finish()

Cleanup.

class reip.blocks.video.effects.OpticalFlow(pyr_scale=0.5, levels=3, winsize=15, iterations=3, poly_n=5, poly_sigma=1.2, flags=0, resize=(256, 256), draw_mag_scale=10, draw=None, **kw)

https://opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/py_video/py_lucas_kanade/py_lucas_kanade.html#dense-optical-flow-in-opencv

init()

Initialize the block.

process(frame, meta)

Process data.

Video Machine Learning

Machine Learning

Disk Monitor

class reip.blocks.diskmonitor.DiskMonitor(root='/', deleter=None, threshold=0.95, padding=0.1, interval=60, **kw)

Monitors the disk usage in a certain directory and will selectively delete files from that directory.

# define the file deleter

@B.diskmonitor.Diskmonitor.deleter
def Diskmon(block, offset=1, skip=3, chunksize=5):
    # select some subset of the files
    files = block.get_files('audio')[offset::skip]
    # randomly delete files in chunks of 5 until
    # under the desired storage limit
    block.delete_while_full(files, chunksize)

# instantiate the block
with reip.Graph():
    Diskmon('/data')

process(*files, meta)

Process data.

Dummy

class reip.blocks.dummy.Array(shape, **kw)

init()

Initialize the block.

process(meta)

Process data.

finish()

Cleanup.

reip.blocks.dummy.SomeArray

alias of reip.blocks.dummy.Array

class reip.blocks.dummy.Op(offset=0, **kw)

process(*data, meta=None)

Process data.

reip.blocks.dummy.SomeTransform

alias of reip.blocks.dummy.Op

class reip.blocks.dummy.TimeBomb(t_minus=5, min_rate=0.001, **kw)

init()

Initialize the block.

process(*xs, meta)

Process data.

class reip.blocks.dummy.TextFile(content, fname=None, **kw)

process(*xs, meta)

Process data.