mcsm_benchs package

Submodules

mcsm_benchs.Benchmark module

mcms-benchs: A toolbox that provides a common framework for benchmarks of multi-component signal processing methods.

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see <https://www.gnu.org/licenses/>.

class mcsm_benchs.Benchmark.Benchmark(task='denoising', methods=None, N=256, Nsub=None, parameters=None, SNRin=None, repetitions=None, signal_ids='all', verbosity=1, parallelize=False, complex_noise=False, obj_fun=None, write_log=True, name=None, description=None, **kwargs)[source]

Bases: object

This class performs a number of tasks for methods comparison. It abstracts a benchmark itself.

add_new_method(methods, parameters=None, perf_func=None)[source]

Add new methods to an existing Benchmark.

Parameters:

methods (_type_) – A dictionary of methods.
parameters (_type_, optional) – If necessary, a dictionary of parameters for the new methods, or new parameters to explore with already benchmarked methods. Defaults to None.

check_methods_output(output, input)[source]

Check that the outputs of the method to benchmark fulfill the required type and shape.

Parameters:

output – Output from the method. The type and shape depends on the task.
input – Input passed to the method to produce output.

Raises:

ValueError – If the output does not comply with the required type and shape
for the selected task. –

static compare_instf_block(signal, method_output, tmin=None, tmax=None, **kwargs)[source]

Compute the instantaneous frequency for multi-component outputs.

Parameters:

signal (numpy.array or Signal) – _description_
method_output (numpy.array) – _description_
tmin (int, optional) – tmin and tmax define a window of the signal to compare SNR. Defaults to None.
tmax (int, optional) – tmin and tmax define a window of the signal to compare SNR. Defaults to None.

Returns:

A dictionary with the IF of each component.

Return type:

dict

static compare_qrf_block(signal, method_output, tmin=None, tmax=None, **kwargs)[source]

Compare the output SNR, i.e. qualitiy reconstruction factor (QRF) in a vectorized way for multi-component outputs.

Parameters:

signal (numpy.array or Signal) – _description_
method_output (numpy.array) – _description_
tmin (int, optional) – tmin and tmax define a window of the signal to compare SNR. Defaults to None.
tmax (int, optional) – tmin and tmax define a window of the signal to compare SNR. Defaults to None.

Returns:

A dictionary with the QRF of each component.

Return type:

dict

static detection_perf_function(original_signal, detection_output, **kwargs)[source]: Performance function for detection. Returns the boolean output of the detection methods.

dic2df(mydic)[source]

Get a pandas DataFrame from a nested dictionary.

Parameters:: mydic (dict) – A nested dictionary of results coming from a Benchmark object.
Returns:: A DataFrame with a column per-level in the dictionary.
Return type:: DataFrame

generate_noise()[source]

Generate a matrix of noise realizations of shape [self.repetitions,self.N]. One realization per row.

Returns:: Matrix with noise realizations.
Return type:: numpy.array

get_results_as_df(results=None)[source]

Get a pandas DataFrame object with the results of the benchmark.

Parameters:

results (dict, optional) – Nested dictionary with the results of the
None. (benchmark. Defaults to)

Returns:

Returns a pandas DataFrame with the results.

Return type:

DataFrame

inner_loop(benchmark_parameters, timer=False)[source]

Main loop of the Benchmark.

Parameters:

benchmark_parameters (tuple) – Tuple or list with the parameters of the benchmark.
timer (bool) – If true, measures the time of execution.

Returns:

Return a numpy array, the shape of which depends on the selected task.

Return type:

narray

input_parsing(task, methods, N, Nsub, parameters, SNRin, repetitions, signal_ids, verbosity, parallelize, complex_noise, obj_fun, **kwargs)[source]

Parse input parameters of the constructor of class Benchmark.

Parameters:

task (str, optional) – The task to test the methods. Defaults to ‘denoising’.
methods (dict, optional) – A dictionary of functions. Defaults to None.
N (int, optional) – Lengths of the signals. Defaults to 256.
parameters (dict, optional) – A dictionary of parameters for the methods
dictionary. (to run. The keys of this dictionary must be same as the methods)
None. (methods.Defaults to)
SNRin (tuple, optional) – List or tuple with the SNR values.
None.
repetitions (int, optional) – Number of times each method is applied for
SNR. (each value of)
the (This value is the number of noise realizations that are used to assess)
None.
signal_ids (tuple, optional) – Tuple or list of the signal ids from the
'all'. (SignalBank class. Defaults to)
verbosity (int, optional) – Number from 0 to 4. It determines the number of
benchmarking (messages passed to the console informing the progress of the)
1. (process. Defaults to)
parallelize (bool, optional) – If True, tries to run the process in parallel.
False. (Defaults to)

Raises:

ValueError – If any parameter is not correctly parsed.

static load_benchmark(filename, **kwargs)[source]

Load a Benchmark object from a file saved using the class method.

Parameters:: filename (str) – A path to the saved benchmark.
Returns:: A Benchmark object.
Return type:: Benchmark

run()[source]

Run the benchmark.

Returns:: Returns nested dictionaries with the results of the benchmark.
Return type:: dict

run_test()[source]

save_to_file(filename=None)[source]

Save the results to a binary file that encodes the benchmark object. Notice that the methods associated with the benchmark, not being pickable objects, are NOT saved.

Parameters:: filename (str, optional) – Path and filename. Defaults to None.
Returns:: True if the file was successfully created.
Return type:: bool

set_objective_function(task)[source]: Set the performance function for the selected task (future tasks could easily add new performance functions)

set_results_dict()[source]: Initializes a dictionary where the results are going to be saved later.

static sigmerge(x1, noise, snr, tmin=None, tmax=None, return_noise=False)[source]

Merge a signal and a noise realization with a given SNR (in dB).

Parameters:

x1 (_type_) – _description_
noise (_type_) – _description_
ratio (_type_) – _description_
tmin (_type_, optional) – _description_. Defaults to None.
tmax (_type_, optional) – _description_. Defaults to None.
return_noise (bool, optional) – _description_. Defaults to False.

Returns:

_description_

Return type:

_type_

Example

>>> import mcsm_benchs.Benchmark
>>> import mcsm_benchs.SignalBank
>>> sb = SignalBank(N=1024)
>>> signal = sb.signal_linear_chirp()
>>> noise = np.random.randn(1024,)
>>> snr = 10 # In dB
>>> mixture = Benchmark.sigmerge(signal,noise,snr=snr)

static snr_comparison(x, x_hat, tmin=None, tmax=None, **kwargs)[source]: Quality reconstruction factor for denoising performance characterization.

static sum(bench_a, bench_b)[source]

This function is used to sum to benchmarks by overloading the + operator. Summing benchmark means transfer the results of bench_b to bench_a as long as they only differ on the methods/parameters used.

Parameters:

bench_a (Benchmark) – The first summand.
bench_b (Benchmark) – The second summand.

Returns:

A Benchmark with the combined methods of bench_a and bench_b.

Return type:

Benchmark

mcsm_benchs.Benchmark.compute_qrf(x, x_hat, tmin=None, tmax=None)[source]: Quality reconstruction factor

mcsm_benchs.Benchmark.corr_comps(x, xest)[source]

Normalized correlation between x and xest.

Parameters:

x (numpy.array) – The origninal noiseless signal.
xest (numpy.array) – An estimation of x.

Returns:

Normalized correlation (between -1 and 1).

Return type:

float

mcsm_benchs.Benchmark.get_args_and_kwargs(params)[source]

mcsm_benchs.Benchmark.mse(x, xest)[source]

Mean square error performance function.

Parameters:

x (numpy.array) – The origninal noiseless signal.
xest (numpy.array) – An estimation of x.

Returns:

Mean square error between x and xest.

Return type:

float

mcsm_benchs.Benchmark.order_components(Xest, X, minormax='max', metric=<function corr_comps>)[source]

This functions receives a multicomponent output Xest of a method and find a correspondance with the original noiseless components X by minimizing (or maximizing) the metric.

Parameters:

Xest (numpy.array) – _description_
X (numpy.array) – _description_
minormax (str, optional) – ‘max’ or ‘min’ according to what is needed from the given metric. Defaults to “max”.
metric (Callable, optional) – A function m = fun(x,xest), with m a real number, and vectors x and xest are the noiseless component x and the estimation of x correspondingly. Defaults to correlation between components.

Returns:

A vector of lenght K, with the correspondence x[0]<->xest[order[0]].

Return type:

numpy.array

mcsm_benchs.MatlabInterface module

mcms-benchs: A toolbox that provides a common framework for benchmarks of multi-component signal processing methods.

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see <https://www.gnu.org/licenses/>.

class mcsm_benchs.MatlabInterface.MatlabInterface(matlab_function_name, add2path=[], matlab_warnings=False)[source]

Bases: object

This class offers an interface between python and Matlab to seamlessly run methods in a Benchmark.

matlab_function(signal, *params)[source]

A wrapper of a Matlab function that receives a signal to process and a variable number of positional arguments.

Parameters:: signal (numpy.ndarray) – A numpy array with a signal.
Returns:: An equivalent array with the outputs of the Matlab function.

pre_parameters(*params)[source]

Cast python types to matlab types before calling the function.

Returns:: A list of matlab types.
Return type:: list

mcsm_benchs.OctaveInterface module

mcms-benchs: A toolbox that provides a common framework for benchmarks of multi-component signal processing methods.

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see <https://www.gnu.org/licenses/>.

class mcsm_benchs.OctaveInterface.OctaveInterface(octave_function_name, add2path=[], matlab_warnings=False)[source]

Bases: object

This class offers an interface between python and Matlab to seamlessly run methods in a Benchmark.

octave_function(signal, *params)[source]

A wrapper of a Matlab function that receives a signal to process and a variable number of positional arguments.

Parameters:: signal (numpy.ndarray) – A numpy array with a signal.
Returns:: An equivalent array with the outputs of the Matlab function.

mcsm_benchs.ResultsInterpreter module

mcms-benchs: A toolbox that provides a common framework for benchmarks of multi-component signal processing methods.

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see <https://www.gnu.org/licenses/>.

class mcsm_benchs.ResultsInterpreter.ResultsInterpreter(a_benchmark)[source]

Bases: object

This class takes a Benchmark-class object to produce a series of plots and tables summarizing the obtained results:

def get_benchmark_as_data_frame(self):

elapsed_time_summary()[source]

Get a DataFrame summarizing the elapsed times of the methods.

Returns:: Pandas DataFrame with a summary of the elapsed times.
Return type:: DataFrame

get_benchmark_as_data_frame(idx=0)[source]

Returns a DataFrame with the raw data produced by the benchmark with the following format:

Method | Parameter | Signal_id | Repetition | SNRin_1 | … | SNRin_n |

Returns:: Raw data of the comparisons.
Return type:: DataFrame

get_csv_files(path, idx=0)[source]

Generates a table of mean results to .md file. Saves a .csv file with the results per signal.

Returns:: String containing the table.
Return type:: str

get_html_figures(df=None, path=None, bars=True, difference=False, varfun=None, ylabel=None, idx=0)[source]: Generate .html interactive plots files with plotly to show online.

get_snr_plot(df, x=None, y=None, hue=None, axis=None, ylabel=None, plot_type='linear', **kwargs)[source]

Generates a Quality Reconstruction Factor (QRF) vs. SNRin plot. The QRF is computed as: QRF = 20 log ( norm(x) / norm(x-x_r)) [dB] where x is the noiseless signal and x_r is de denoised estimation of x.

Parameters:

df (DataFrame) – DataFrame with the results of the simulation.
x (str, optional) – Column name to use as the horizontal axis.
None. (generated. Defaults to)
y (str, optional) – Column name to use as the vertical axis.
None.
hue (str, optional) – Column name with the methods’ name. Defaults to None.
axis (matplotlib.Axes, optional) – The axis object where the plot will be
None.

get_snr_plot_bars(df, x=None, y=None, hue=None, errbar_fun=('ci', 95), axis=None, errbar_params=None)[source]

Generates a Quality Reconstruction Factor (QRF) vs. SNRin barplot. The QRF is computed as:

QRF = 20 log ( norm(x) / norm(x-x_r)) [dB]

where x is the noiseless signal and x_r is de denoised estimation of x.

Parameters:

df (DataFrame) – DataFrame with the results of the simulation.
x (str, optional) – Column name to use as the horizontal axis.
None. (generated. Defaults to)
y (str, optional) – Column name to use as the vertical axis.
None.
hue (str, optional) – Column name with the methods’ name. Defaults to None.
axis (matplotlib.Axes, optional) – The axis object where the plot will be
None.

get_summary_plotlys(df=None, bars=True, difference=False, varfun=None, ylabel=None, idx=0)[source]

Generates interactive plots with plotly.

Returns:: A list with plotly figures.
Return type:: list

get_summary_plots(df_rearr=None, size=(3, 3), savetofile=True, path=None, filter_crit='all', filter_str=None, errbar_fun=('ci', 95), errbar_params=None, ax=None, plot_type='bars', bars=True, magnitude='absolute')[source]

Generates individual performance plots for each signal, displaying the performance measure of all methods for all noise conditions.

Parameters:

size (tuple, optional) – Size (in inches) of the figures. Defaults to (3,3).
savetofile (bool, optional) – Whether to save or not the figures.
True. (Defaults to)
filename (_type_, optional) – Path and file name to save the figures. If None
None. (plots all the methods. Defaults to)
filter_str (_type_, optional) – A string, or a list of strings, to select
None (the methods to plot. If)
None.
filter_crit (str, optional) – A criterion to use the strings passed in
'all' (filter_str. If)
strings (only choose those methods where all the)
'any' (appear in the "Method" column of the resuls DataFrame. If)
column. (select the methods for wich any of the strings appear in the mentioned)
'all'. (Defaults to)
plot_type (str, optional) – _description_. Defaults to ‘lines’.

Returns:

A list with matplotlib figures.

Return type:

list

rearrange_data_frame(results=None, idx=0)[source]

Rearrange DataFrame table for using Seaborn library.

Parameters:

results (DataFrame, optional) – If not None, must receive the DataFrame
None (produced by a Benchmark-class object using get_results_as_df(). If)

:param : :param uses the Benchmark object given to the constructor of the Interpreter.: :param Defaults to None.:

Returns:: Rearranged DataFrame
Return type:: DataFrame

save_report(filename=None, path='results', bars=False, link='', pm_name=None)[source]

This function generates a report of the results given in the Benchmark-class object. The report is saved in a Markdown syntax to be viewed as a .md file, while a .csv file is generated with the results.

Parameters:: filename (str, optional) – Path for saving the report. Defaults to None.

mcsm_benchs.SignalBank module

mcms-benchs: A toolbox that provides a common framework for benchmarks of multi-component signal processing methods.

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see <https://www.gnu.org/licenses/>.

class mcsm_benchs.SignalBank.Signal(array, instf=None, buffer=None, offset=0, strides=None, order=None)[source]

Bases: ndarray

A class that emulates a signal by behaving like a Numpy array for practical purposes but containing information of the signal such as the number of components and their instantaneous frequency (IF). This class is used in two ways:

1. As a way of creating more complicated signals (i.e. with more components) and automatically stablish the number of components and IF by just defining them for the more simple, monocomponent signals. This way, when linearly combining two signals all the information regarding the combined components and their instantaneous frequencies is saved and is accessible from the final signal.

2. As a way of wrap up information of the signal when passed to a method in the Benchmark class. The idea is that methods called by this latter class only receive a signal and parameters for the method. Therefore, by encapsulating the information of the signal in the Signal object, the method can use it while keep treating the received signal as a regular numpy array.

A Signal class object has four attributes that differentiate it from a regular numpy array: 1. comps: A list with each of the individual components combined to produce the signal. 2. insft: A list with each of the individual components’ instantenous frequency. The length of this list is the same as the comps list. 3. ncomps: A numpy array indicating the number of components present in each time sample of the signal. 4. total_comps: The total amount of components present within the duration of the signal. This is simply the length of comps.

def add_comp(self, new_comp, \*\*kwargs)

def add_instf(self, new_instf, \*\*kwargs)

add_comp(new_comp, **kwargs)[source]

Add a new component, potentially with its instantaneous frequency. This latter can be added later.

Parameters:

new_comp (numpy.ndarray) – New component to add.
instf (numpy.ndarray) – The instantaneous frequency corresponding to new_comp

add_instf(new_instf, **kwargs)[source]

Add a new instantaneous frequency.

Parameters:: new_instf (numpy.ndarray) – A vector with the instantaneous frequency.

component_counter()[source]: This functions counts the number of components per time sample, and set the corresponding attribute self._ncomps.

property comps

get_info()[source]

property instf

property ncomps

std(axis=None, dtype=None, out=None, ddof=0, keepdims=False, *, where=True)[source]

Returns the standard deviation of the array elements along given axis.

Refer to numpy.std for full documentation.

mcsm_benchs.benchmark_utils module

mcms-benchs: A common framework for benchmarks of multi-component signal processing methods.

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see <https://www.gnu.org/licenses/>.

class mcsm_benchs.benchmark_utils.MethodTemplate[source]

Bases: ABC

An abstract class for new methods to add in a benchmark.

get_parameters()[source]

property id

abstract method()[source]

property task

mcsm_benchs package

Submodules

mcsm_benchs.Benchmark module

mcsm_benchs.MatlabInterface module

mcsm_benchs.OctaveInterface module

mcsm_benchs.ResultsInterpreter module

mcsm_benchs.SignalBank module

mcsm_benchs.benchmark_utils module

Module contents