Custom Pipelines#

The pipeline module provides a general chunked pipeline entry point (massphoto.process()) and workflow helpers.

By default, parallel outputs carry an analysis_record in DataFrame.attrs (attach_record=True in high-level APIs). Use massphoto.resolve_record() to turn that record back into a defaults-aware replay callable.

Pipeline Type Markers#

When assembling custom pipelines with massphoto.process(), function parameters use type annotations to declare what data representation they consume (see Define a custom detection function in the tutorial gallery):

massphoto.NativeMovie: NewType('NativeMovie', np.ndarray)#

Annotates parameters that expect raw camera movie frames (frames, y, x). The chunked pipeline runner passes slices of the original input movie to these parameters.

massphoto.RatiometricMovie: NewType('RatiometricMovie', np.ndarray)#

Annotates parameters that expect background-subtracted ratiometric images. When requested, the pipeline ensures ratiometric frames are generated before invoking the step.

massphoto.EventList: NewType('EventList', pd.DataFrame)#

Annotates functions returning or processing detected events in a pandas.DataFrame containing required spatial coordinates (x, y) and frame indices (frame).

Module Reference#

massphoto.pipeline.resolve_record(obj, *, apply_defaults=False)[source]#

Resolve an analysis record into defaults overrides and a replay callable.

Note

Available as both massphoto.resolve_record and massphoto.pipeline.resolve_record.

Parameters:

  • obj (dict or pd.DataFrame) – Either an analysis_record dictionary or a DataFrame carrying one in obj.attrs["analysis_record"].

  • apply_defaults (bool, default=False) – If True, apply the resolved defaults state globally via massphoto.defaults.

Returns:

(defaults_overrides, replay_callable) where:

  • defaults_overrides is the merged defaults state inferred from the record and its origin kwargs.

  • replay_callable is a defaults-wrapped callable for the recorded origin (process, detect, or fit), preloaded with defaults_overrides.

Return type:

tuple[dict, Callable]

Raises:

  • TypeError – If obj is neither a record dict nor a DataFrame with dict attrs['analysis_record'], or if origin_kwargs is not a dict.

  • ValueError – If required record keys are missing, the schema version is unsupported, or the record origin is unknown. The supported schema version is 2.

massphoto.pipeline.process(movie, steps, *, events=None, movie_template=None, movie_normalize_frame_means=False, movie_correct_drift=False, return_ratiometric=False, ratiometric_args=None, ratiometric_func=<function calc_ratiometric>, chunk_overlap=-1, chunk_size=1000, postprocess=(), scheduler=None, progress_bar=True, attach_record=True, compute_movie_metrics=False)[source]#

Run a chunked, parallel processing pipeline over a movie.

Note

Available as both massphoto.process and massphoto.pipeline.process.

Each step is a (callable, kwargs) pair. Inputs for each callable are inferred from type annotations (NativeMovie, RatiometricMovie, EventList/pd.DataFrame). Optional precomputed events can be provided to seed the pipeline. movie_template is the global native reference image used by movie-level preprocessing. When movie_normalize_frame_means=True, each native frame is scaled so its mean matches movie_template.mean(). When movie_correct_drift=True, the same template is passed to drift correction. If needed and not supplied, the template is computed as the mean image of the input movie.

The optional postprocess callables are applied to the final (non-chunked) DataFrame after compute.

When attach_record=True (default), the returned DataFrame stores an analysis_record in DataFrame.attrs. The record captures enough provenance to replay the analysis with equivalent defaults context:

  • defaults_state: a copy of massphoto.defaults at runtime.

  • origin and origin_kwargs: the entry-point function name and the captured call kwargs.

  • movie_info: input movie shape and dtype.

For tuple outputs (return_ratiometric=True), the record is attached to the DataFrame element. Use resolve_record() to reconstruct a replay callable from the stored record.

Parameters: