tfma.utils.CombineFnWithModels

Abstract class for CombineFns that need the shared models.

tfma.utils.CombineFnWithModels(
    model_loaders: Dict[str, tfma.types.ModelLoader]
)

Methods

`add_input`

add_input(
    mutable_accumulator, element, *args, **kwargs
)

Return result of folding element into accumulator.

CombineFn implementors must override add_input.

Args
`mutable_accumulator`	the current accumulator, may be modified and returned for efficiency
`element`	the element to add, should not be mutated
`*args`	Additional arguments and side inputs.
`**kwargs`	Additional arguments and side inputs.

`add_inputs`

add_inputs(
    mutable_accumulator, elements, *args, **kwargs
)

Returns the result of folding each element in elements into accumulator.

This is provided in case the implementation affords more efficient bulk addition of elements. The default implementation simply loops over the inputs invoking add_input for each one.

Args
`mutable_accumulator`	the current accumulator, may be modified and returned for efficiency
`elements`	the elements to add, should not be mutated
`*args`	Additional arguments and side inputs.
`**kwargs`	Additional arguments and side inputs.

`apply`

apply(
    elements, *args, **kwargs
)

Returns result of applying this CombineFn to the input values.

Args
`elements`	the set of values to combine.
`*args`	Additional arguments and side inputs.
`**kwargs`	Additional arguments and side inputs.

`compact`

compact(
    accumulator, *args, **kwargs
)

Optionally returns a more compact representation of the accumulator.

This is called before an accumulator is sent across the wire, and can be useful in cases where values are buffered or otherwise lazily kept unprocessed when added to the accumulator. Should return an equivalent, though possibly modified, accumulator.

By default returns the accumulator unmodified.

Args
`accumulator`	the current accumulator
`*args`	Additional arguments and side inputs.
`**kwargs`	Additional arguments and side inputs.

`create_accumulator`

create_accumulator(
    *args, **kwargs
)

Return a fresh, empty accumulator for the combine operation.

Args
`*args`	Additional arguments and side inputs.
`**kwargs`	Additional arguments and side inputs.

`default_label`

default_label()

`default_type_hints`

default_type_hints()

`display_data`

display_data()

Returns the display data associated to a pipeline component.

It should be reimplemented in pipeline components that wish to have static display data.

Returns

Returns
Dict[str, Any]: A dictionary containing `key:value` pairs. The value might be an integer, float or string value; a :class:`DisplayDataItem` for values that have more data (e.g. short value, label, url); or a :class:`HasDisplayData` instance that has more display data that should be picked up. For example:: { 'key1': 'string_value', 'key2': 1234, 'key3': 3.14159265, 'key4': DisplayDataItem('apache.org', url='http://apache.org'), 'key5': subComponent }

Dict[str, Any]: A dictionary containing key:value pairs. The value might be an integer, float or string value; a :class:DisplayDataItem for values that have more data (e.g. short value, label, url); or a :class:HasDisplayData instance that has more display data that should be picked up. For example::

{ 'key1': 'string_value', 'key2': 1234, 'key3': 3.14159265, 'key4': DisplayDataItem('apache.org', url='http://apache.org'), 'key5': subComponent }

`extract_output`

extract_output(
    accumulator, *args, **kwargs
)

Return result of converting accumulator into the output value.

Args
`accumulator`	the final accumulator value computed by this CombineFn for the entire input key or PCollection. Can be modified for efficiency.
`*args`	Additional arguments and side inputs.
`**kwargs`	Additional arguments and side inputs.

`for_input_type`

for_input_type(
    input_type
)

Returns a specialized implementation of self, if it exists.

Otherwise, returns self.

Args
`input_type`	the type of input elements.

`from_callable`

@staticmethod
from_callable(
    fn
)

`from_runner_api`

@classmethod
from_runner_api(
    fn_proto: beam_runner_api_pb2.FunctionSpec, context: 'PipelineContext'
) -> RunnerApiFnT

Converts from an FunctionSpec to a Fn object.

Prefer registering a urn with its parameter type and constructor.

`get_accumulator_coder`

get_accumulator_coder()

`get_type_hints`

get_type_hints()

Gets and/or initializes type hints for this object.

If type hints have not been set, attempts to initialize type hints in this order:

Using self.default_type_hints().
Using self.class type hints.

`maybe_from_callable`

@staticmethod
maybe_from_callable(
    fn, has_side_inputs=True
)

`merge_accumulators`

merge_accumulators(
    accumulators, *args, **kwargs
)

Returns the result of merging several accumulators to a single accumulator value.

Args
`accumulators`	the accumulators to merge. Only the first accumulator may be modified and returned for efficiency; the other accumulators should not be mutated, because they may be shared with other code and mutating them could lead to incorrect results or data corruption.
`*args`	Additional arguments and side inputs.
`**kwargs`	Additional arguments and side inputs.

`register_pickle_urn`

@classmethod
register_pickle_urn(
    pickle_urn
)

Registers and implements the given urn via pickling.

`register_urn`

@classmethod
register_urn(
    urn, parameter_type, fn=None
)

Registers a urn with a constructor.

For example, if 'beam:fn:foo' had parameter type FooPayload, one could write RunnerApiFn.register_urn('bean:fn:foo', FooPayload, foo_from_proto) where foo_from_proto took as arguments a FooPayload and a PipelineContext. This function can also be used as a decorator rather than passing the callable in as the final parameter.

A corresponding to_runner_api_parameter method would be expected that returns the tuple ('beam:fn:foo', FooPayload)

`setup`

View source

setup()

Called to prepare an instance for combining.

This method can be useful if there is some state that needs to be loaded before executing any of the other methods. The resources can then be disposed of in CombineFn.teardown.

If you are using Dataflow, you need to enable Dataflow Runner V2 before using this feature.

Args
`*args`	Additional arguments and side inputs.
`**kwargs`	Additional arguments and side inputs.

`teardown`

teardown(
    *args, **kwargs
)

Called to clean up an instance before it is discarded.

If you are using Dataflow, you need to enable Dataflow Runner V2 before using this feature.

Args
`*args`	Additional arguments and side inputs.
`**kwargs`	Additional arguments and side inputs.

`to_runner_api`

to_runner_api(
    context: 'PipelineContext'
) -> beam_runner_api_pb2.FunctionSpec

Returns an FunctionSpec encoding this Fn.

Prefer overriding self.to_runner_api_parameter.

`to_runner_api_parameter`

to_runner_api_parameter(
    context
)

`with_input_types`

with_input_types(
    *arg_hints, **kwarg_hints
) -> WithTypeHintsT

`with_output_types`

with_output_types(
    *arg_hints, **kwarg_hints
) -> WithTypeHintsT