{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# `mcsm-benchs`: Creating benchmarks of MCS Methods\n", "\n", "We introduce a public, open-source, Python-based toolbox for benchmarking multi-component signal analysis methods, implemented either in Python or Matlab.\n", "\n", "The goal of this toolbox is providing the signal-processing community with a common framework that allows researcher-independent comparisons between methods and favors reproducible research.\n", "\n", "With the purpose of making this toolbox more useful, the methods to compare, the tests, the signal generation code and the performance evaluation functions were conceived as different modules, so that one can modify them independently. The only restriction this pose is that the methods should satisfy some requirements regarding the shape of their input and output parameters.\n", "\n", "On the one hand, the tests and the performance evaluation functions, are encapsulated in the class `Benchmark`. On the other hand, the signals used in this benchmark are generated by the methods in the class `SignalBank`.\n", "\n", "In order to compare different methods with possibly different parameters, we need to set up a few things before running the benchmark. A `Benchmark` object receives some input parameters to configure the test:\n", "- `task`: This could be `'denoising'` or `'detection'`,`'component_denoising'` or `'inst_frequency'`. The first one compute the quality reconstruction factor (QRF) using the output of the method, whereas the second simply consist in detecting whether a signal is present or not. Finally, `'component_denoising'` compares the QRF component wise, and '`inst_frequency'` computes the mean squared error between estimations of the instantaneous frequency\n", "- `N`: The length of the simulation, i.e. how many samples should the signals have.\n", "- `methods`: A dictionary of methods. Each entry of this dictionary corresponds to the function that implements each of the desired methods.\n", "- `parameters`: A dictionary of parameters. Each entry of this dictionary corresponds to iterator with positional and/or keyword arguments. In order to know which parameters should be passed to each method, the keys of this dictionary should be the same as those corresponding to the individual methods in the corresponding dictionary. An example of this is showed in below.\n", "- `SNRin`: A list or tuple of values of SNR to test.\n", "- `repetitions`: The number of times the experiment should be repeated with different realizations of noise.\n", "- `signal_ids`: A list of signal ids (corresponding to the names of the signal in the class '`SignalBank`') can be passed here in order to test the methods on those signals. Optionally, the user can pass a dictionary where each key is used as an identifier, and the corresponding value can be a numpy array with a personalized signal." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## A dummy test \n", "First let us define a dummy method for testing. Methods should receive a numpy array with shape ```(N,)``` where `N` is the number of time samples of the signal. Additionally, they can receive any number of positional or keyword arguments to allow testing different combinations of input parameters. The shape of the output depends on the task (signal denoising or detection). So the recommended signature of a method should be the following:\n", "\n", " `output = a_method(noisy_signal, *args, **kwargs) `.\n", "\n", "If one set `task='denoising'`, `output` shoud be a `(N,)` numpy array, i.e. the same shape as the input parameter `noisy_signal`, whereas if `task='detection'`, the output should be boolean (`0` or `False` for no signal, and `1` or `True` otherwise).\n", "\n", "After this, we need to create a *dictionary of methods* to pass the `Benchmark` object at the moment of instantiation." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "from numpy import pi as pi\n", "import pandas as pd\n", "from matplotlib import pyplot as plt\n", "from mcsm_benchs.Benchmark import Benchmark\n", "from mcsm_benchs.ResultsInterpreter import ResultsInterpreter\n", "from mcsm_benchs.SignalBank import SignalBank\n", "from utils import spectrogram_thresholding, get_stft" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Creating a dictionary of methods\n", "\n", "Let's create a dictionary of methods to benchmark. As as example, we will compare two strategies for spectrogram thresholding.\n", "The first one is hard thresholding, in which the thresholding function is defined as:\n", "The second one is soft thresholding, here defined as:\n", "\n", "These two approaches are implemented in the python function ```thresholding(signal, lam, fun='hard')``` function, which receives a signal to clean, a positional argument ```lam``` and a keyword argument ```fun``` that can be either ```hard``` or ```soft```.\n", " \n", "Our dictionary of methods will consist then in two methods: hard thresholding and soft thresholding.\n", "For both approaches, let's use a value of ```lam=1.0``` for now." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "\n", "def method_1(noisy_signal, *args, **kwargs):\n", " # If additional input parameters are needed, they can be passed in a tuple using \n", " # *args or **kwargs and then parsed.\n", " xr = spectrogram_thresholding(noisy_signal,1.0,fun='hard')\n", " return xr\n", "\n", "def method_2(noisy_signal, *args, **kwargs):\n", " # If additional input parameters are needed, they can be passed in a tuple using \n", " # *args or **kwargs and then parsed.\n", " xr = spectrogram_thresholding(noisy_signal,2.0,fun='soft') \n", " return xr\n", "\n", "# Create a dictionary of the methods to test.\n", "my_methods = {\n", " 'Method 1': method_1, \n", " 'Method 2': method_2,\n", " }" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The variable `params` in the example above allows us to pass some parameters to our method. This would be useful for testing a single method with several combination of input parameters. In order to do this, we should give the `Benchmark` object a *dictionary of parameters*. An example of this functionality is showed in the next section. For now, lets set the input parameter `parameters = None`." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we are ready to instantiate a `Benchmark` object and run a test using the proposed methods and parameters. The benchmark constructor receives a name of a task (which defines the performance function of the test), a dictionary of the methods to test, the desired length of the signals used in the simulation, a dictionary of different parameters that should be passed to the methods, an array with different values of SNR to test, and the number of repetitions that should be used for each test. Once the object is created, use the class method `run_test()` to start the experiments.\n", "\n", "*Remark 1: You can use the ```verbosity``` parameter to show less or more messages during the progress of the experiments. There are 6 levels of verbosity, from ```verbosity=0``` (indicate just the start and the end of the experiments) to ```verbostiy = 5``` (show each method and parameter progress)*\n", "\n", "*Remark 2: Parallelize the experiments is also possible by passing the parameter ```parallelize = True```.*" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Method run_test() will be deprecated in newer versions. Use run() instead.\n", "Running benchmark...\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "100%|██████████| 2/2 [00:00<00:00, 2.17it/s]\n", "100%|██████████| 2/2 [00:00<00:00, 2.19it/s]\n" ] }, { "data": { "text/plain": [ "True" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "benchmark = Benchmark(task = 'denoising',\n", " methods = my_methods,\n", " N = 256, \n", " SNRin = [10,20], \n", " repetitions = 1000,\n", " signal_ids=['LinearChirp', 'CosChirp',],\n", " verbosity=0, \n", " parallelize=False)\n", " \n", "benchmark.run_test() # Run the test. my_results is a dictionary with the results for each of the variables of the simulation.\n", "benchmark.save_to_file('saved_benchmark') # Save the benchmark to a file." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "same_benchmark = Benchmark.load_benchmark('saved_benchmark') # Load the benchmark from a file." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we have the results of the test in a nested dictionary called `my_results`. In order to get the results in a human-readable way using a `DataFrame`, and also for further analysis and reproducibility, we can use the class method `get_results_as_df()`." ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "application/vnd.microsoft.datawrangler.viewer.v0+json": { "columns": [ { "name": "index", "rawType": "int64", "type": "integer" }, { "name": "Method", "rawType": "object", "type": "string" }, { "name": "Parameter", "rawType": "object", "type": "string" }, { "name": "Signal_id", "rawType": "object", "type": "string" }, { "name": "Repetition", "rawType": "int64", "type": "integer" }, { "name": "10", "rawType": "float64", "type": "float" }, { "name": "20", "rawType": "float64", "type": "float" } ], "conversionMethod": "pd.DataFrame", "ref": "ae3b3c8c-34f0-457b-bc25-fd7cdf928093", "rows": [ [ "2000", "Method 1", "((), {})", "CosChirp", "0", "11.886835659114459", "22.37715108264989" ], [ "2001", "Method 1", "((), {})", "CosChirp", "1", "12.722812880140566", "22.764551803585363" ], [ "2002", "Method 1", "((), {})", "CosChirp", "2", "12.397019845144467", "22.36957426399826" ], [ "2003", "Method 1", "((), {})", "CosChirp", "3", "11.548921986599542", "22.221890113454723" ], [ "2004", "Method 1", "((), {})", "CosChirp", "4", "12.083445022222206", "22.151974820017728" ], [ "2005", "Method 1", "((), {})", "CosChirp", "5", "11.539915499209435", "22.53463399279379" ], [ "2006", "Method 1", "((), {})", "CosChirp", "6", "11.770607241799407", "21.407158778601953" ], [ "2007", "Method 1", "((), {})", "CosChirp", "7", "12.000421885788935", "22.195617040199814" ], [ "2008", "Method 1", "((), {})", "CosChirp", "8", "12.652924550241874", "21.951620063076483" ], [ "2009", "Method 1", "((), {})", "CosChirp", "9", "11.817618948427782", "21.88709345740556" ], [ "2010", "Method 1", "((), {})", "CosChirp", "10", "11.900853230162847", "21.977302024204427" ], [ "2011", "Method 1", "((), {})", "CosChirp", "11", "11.001221186309408", "21.35456323414907" ], [ "2012", "Method 1", "((), {})", "CosChirp", "12", "12.907874381154327", "21.957742911019867" ], [ "2013", "Method 1", "((), {})", "CosChirp", "13", "12.255905772305237", "21.983145390884744" ], [ "2014", "Method 1", "((), {})", "CosChirp", "14", "11.957253776232415", "21.674801583988533" ], [ "2015", "Method 1", "((), {})", "CosChirp", "15", "12.440681429480485", "22.41920624855024" ], [ "2016", "Method 1", "((), {})", "CosChirp", "16", "12.753937824574304", "22.37677643772023" ], [ "2017", "Method 1", "((), {})", "CosChirp", "17", "12.188847336244805", "22.456816318445924" ], [ "2018", "Method 1", "((), {})", "CosChirp", "18", "11.61633170042943", "21.38442048453278" ], [ "2019", "Method 1", "((), {})", "CosChirp", "19", "11.29460659033149", "21.617649342457348" ], [ "2020", "Method 1", "((), {})", "CosChirp", "20", "11.878664288767396", "21.949846904186717" ], [ "2021", "Method 1", "((), {})", "CosChirp", "21", "12.227231021522567", "22.16163076673108" ], [ "2022", "Method 1", "((), {})", "CosChirp", "22", "12.328159171861113", "21.810523267392924" ], [ "2023", "Method 1", "((), {})", "CosChirp", "23", "11.830928625729907", "22.043764377267667" ], [ "2024", "Method 1", "((), {})", "CosChirp", "24", "11.757553405878731", "22.086809518009606" ], [ "2025", "Method 1", "((), {})", "CosChirp", "25", "11.602778809406269", "22.162922728620664" ], [ "2026", "Method 1", "((), {})", "CosChirp", "26", "11.322351315128136", "21.229056573057733" ], [ "2027", "Method 1", "((), {})", "CosChirp", "27", "11.675910164139932", "21.705963888256747" ], [ "2028", "Method 1", "((), {})", "CosChirp", "28", "12.18074812360398", "21.468522364267073" ], [ "2029", "Method 1", "((), {})", "CosChirp", "29", "11.566105566657379", "22.51018186552017" ], [ "2030", "Method 1", "((), {})", "CosChirp", "30", "11.809897470131347", "21.943586972996563" ], [ "2031", "Method 1", "((), {})", "CosChirp", "31", "12.414582697613046", "22.23237944545984" ], [ "2032", "Method 1", "((), {})", "CosChirp", "32", "11.842205096048755", "21.8652492247759" ], [ "2033", "Method 1", "((), {})", "CosChirp", "33", "11.819630726493067", "22.317041509046298" ], [ "2034", "Method 1", "((), {})", "CosChirp", "34", "11.805780763174152", "22.167551829970776" ], [ "2035", "Method 1", "((), {})", "CosChirp", "35", "11.828109546201816", "21.91305629389603" ], [ "2036", "Method 1", "((), {})", "CosChirp", "36", "11.90839517056267", "22.852462761172365" ], [ "2037", "Method 1", "((), {})", "CosChirp", "37", "12.14548448227617", "22.38907315081512" ], [ "2038", "Method 1", "((), {})", "CosChirp", "38", "12.143874598997845", "22.10592604370714" ], [ "2039", "Method 1", "((), {})", "CosChirp", "39", "11.421471898983084", "21.69739089982094" ], [ "2040", "Method 1", "((), {})", "CosChirp", "40", "11.825547173046452", "22.010842237646063" ], [ "2041", "Method 1", "((), {})", "CosChirp", "41", "11.810841416294288", "21.746285428594998" ], [ "2042", "Method 1", "((), {})", "CosChirp", "42", "12.447061622080934", "21.73764559882136" ], [ "2043", "Method 1", "((), {})", "CosChirp", "43", "11.240800320186642", "22.14371900944532" ], [ "2044", "Method 1", "((), {})", "CosChirp", "44", "12.015941591158622", "21.774523385501507" ], [ "2045", "Method 1", "((), {})", "CosChirp", "45", "12.101558158573786", "22.66403378633359" ], [ "2046", "Method 1", "((), {})", "CosChirp", "46", "12.534344263609299", "22.24124872924367" ], [ "2047", "Method 1", "((), {})", "CosChirp", "47", "12.411271239591004", "21.9361978584311" ], [ "2048", "Method 1", "((), {})", "CosChirp", "48", "12.576631643326838", "22.208336530986283" ], [ "2049", "Method 1", "((), {})", "CosChirp", "49", "12.503972921230329", "21.93850644154994" ] ], "shape": { "columns": 6, "rows": 4000 } }, "text/html": [ "
\n", " | Method | \n", "Parameter | \n", "Signal_id | \n", "Repetition | \n", "10 | \n", "20 | \n", "
---|---|---|---|---|---|---|
2000 | \n", "Method 1 | \n", "((), {}) | \n", "CosChirp | \n", "0 | \n", "11.886836 | \n", "22.377151 | \n", "
2001 | \n", "Method 1 | \n", "((), {}) | \n", "CosChirp | \n", "1 | \n", "12.722813 | \n", "22.764552 | \n", "
2002 | \n", "Method 1 | \n", "((), {}) | \n", "CosChirp | \n", "2 | \n", "12.397020 | \n", "22.369574 | \n", "
2003 | \n", "Method 1 | \n", "((), {}) | \n", "CosChirp | \n", "3 | \n", "11.548922 | \n", "22.221890 | \n", "
2004 | \n", "Method 1 | \n", "((), {}) | \n", "CosChirp | \n", "4 | \n", "12.083445 | \n", "22.151975 | \n", "
... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "
1995 | \n", "Method 2 | \n", "((), {}) | \n", "LinearChirp | \n", "995 | \n", "6.838226 | \n", "17.240144 | \n", "
1996 | \n", "Method 2 | \n", "((), {}) | \n", "LinearChirp | \n", "996 | \n", "6.713695 | \n", "18.924030 | \n", "
1997 | \n", "Method 2 | \n", "((), {}) | \n", "LinearChirp | \n", "997 | \n", "6.418767 | \n", "18.474438 | \n", "
1998 | \n", "Method 2 | \n", "((), {}) | \n", "LinearChirp | \n", "998 | \n", "6.711867 | \n", "18.170891 | \n", "
1999 | \n", "Method 2 | \n", "((), {}) | \n", "LinearChirp | \n", "999 | \n", "7.773640 | \n", "19.012253 | \n", "
4000 rows × 6 columns
\n", "\n", " | Average time (s) | \n", "Std | \n", "
---|---|---|
CosChirp-Method 1-((1.0,), {'fun': 'hard'}) | \n", "0.000252 | \n", "0.000106 | \n", "
CosChirp-Method 1-((2.0,), {'fun': 'hard'}) | \n", "0.000196 | \n", "0.000015 | \n", "
CosChirp-Method 1-((3.0,), {'fun': 'hard'}) | \n", "0.000198 | \n", "0.000019 | \n", "
CosChirp-Method 2-((1.0,), {'fun': 'soft'}) | \n", "0.000207 | \n", "0.000015 | \n", "
CosChirp-Method 2-((2.0,), {'fun': 'soft'}) | \n", "0.000178 | \n", "0.000007 | \n", "
CosChirp-Method 2-((3.0,), {'fun': 'soft'}) | \n", "0.000253 | \n", "0.000024 | \n", "
LinearChirp-Method 1-((1.0,), {'fun': 'hard'}) | \n", "0.000169 | \n", "0.000015 | \n", "
LinearChirp-Method 1-((2.0,), {'fun': 'hard'}) | \n", "0.000233 | \n", "0.000087 | \n", "
LinearChirp-Method 1-((3.0,), {'fun': 'hard'}) | \n", "0.000220 | \n", "0.000054 | \n", "
LinearChirp-Method 2-((1.0,), {'fun': 'soft'}) | \n", "0.000209 | \n", "0.000015 | \n", "
LinearChirp-Method 2-((2.0,), {'fun': 'soft'}) | \n", "0.000210 | \n", "0.000011 | \n", "
LinearChirp-Method 2-((3.0,), {'fun': 'soft'}) | \n", "0.000226 | \n", "0.000036 | \n", "
\n", " | SNRin | \n", "Method | \n", "Parameter | \n", "Signal_id | \n", "Repetition | \n", "QRF | \n", "
---|---|---|---|---|---|---|
0 | \n", "10 | \n", "Method 1 | \n", "((1.0,), {'fun': 'hard'}) | \n", "CosChirp | \n", "0 | \n", "11.886836 | \n", "
1 | \n", "10 | \n", "Method 1 | \n", "((1.0,), {'fun': 'hard'}) | \n", "CosChirp | \n", "1 | \n", "12.722813 | \n", "
2 | \n", "10 | \n", "Method 1 | \n", "((1.0,), {'fun': 'hard'}) | \n", "CosChirp | \n", "2 | \n", "12.397020 | \n", "
3 | \n", "10 | \n", "Method 1 | \n", "((1.0,), {'fun': 'hard'}) | \n", "CosChirp | \n", "3 | \n", "11.548922 | \n", "
4 | \n", "10 | \n", "Method 1 | \n", "((1.0,), {'fun': 'hard'}) | \n", "CosChirp | \n", "4 | \n", "12.083445 | \n", "
... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "
535 | \n", "30 | \n", "Method 2 | \n", "((3.0,), {'fun': 'soft'}) | \n", "LinearChirp | \n", "10 | \n", "25.606940 | \n", "
536 | \n", "30 | \n", "Method 2 | \n", "((3.0,), {'fun': 'soft'}) | \n", "LinearChirp | \n", "11 | \n", "25.779403 | \n", "
537 | \n", "30 | \n", "Method 2 | \n", "((3.0,), {'fun': 'soft'}) | \n", "LinearChirp | \n", "12 | \n", "25.338387 | \n", "
538 | \n", "30 | \n", "Method 2 | \n", "((3.0,), {'fun': 'soft'}) | \n", "LinearChirp | \n", "13 | \n", "23.843373 | \n", "
539 | \n", "30 | \n", "Method 2 | \n", "((3.0,), {'fun': 'soft'}) | \n", "LinearChirp | \n", "14 | \n", "25.568327 | \n", "
540 rows × 6 columns
\n", "