mcsm-benchs: Benchmarking methods for instantaneous frequency estimation
[1]:
import numpy as np
from numpy import pi as pi
import scipy.signal as sg
import pandas as pd
from matplotlib import pyplot as plt
from mcsm_benchs.Benchmark import Benchmark
from mcsm_benchs.ResultsInterpreter import ResultsInterpreter
from mcsm_benchs.SignalBank import SignalBank
from utils import get_stft
Creating a dictionary of signals
We can use the SignalBank class to generate a dictionary of signals to study. We are going to use three signals: 1) a linear chirp, 2) a cosenoidal chirp and 3) an exponential chirp. Below we can see how to generate the signals as well as their spectrograms, which show the trajectories of the instantaneous frequencies giving name to the signals.
We use the option return_signal=True, so that the signals generated by the SignalBank are objects of the Signal class, which behave like a regular numpy array, but include additional information of the generated signals, such as the instantaneous frequency of each signal component.
[2]:
# Create a dictionary of signals:
N = 128
sb = SignalBank(N=N, return_signal=True)
signal_1 = sb.signal_linear_chirp()
signal_2 = sb.signal_cos_chirp()
signal_3 = sb.signal_exp_chirp()
# Create a dictionary of signals for later.
signals = {'linear_chirp':signal_1, 'cos_chirp':signal_2, 'exp_chirp':signal_3}
# Display the spectrograms of the signals.
stft1 = get_stft(signal_1)
stft2 = get_stft(signal_2)
stft3 = get_stft(signal_3)
fig, axs = plt.subplots(1,3, sharey=True)
axs[0].imshow(np.abs(stft1)**2, origin='lower')
axs[1].imshow(np.abs(stft2)**2, origin='lower')
axs[2].imshow(np.abs(stft3)**2, origin='lower')
[2]:
<matplotlib.image.AxesImage at 0x7167dccb4470>
Creating a dictionary of methods
We will define two methods to estimate the instantaneous frequency of the previously created signals. Notice that all signals have only one component, which simplifies our methods, but any method outputting a ``list`` of estimated instantaneous frequencies (one per component) can be benchmarked. The benchmarking procedure takes care of matching the output of the method with the most similar component from the signal in order to compute the performance metric of each IF estimation.
An instantaneous frequency estimation consists in an array of the same length of the signal with normalized frequency (between \(0\) and \(0.5\)). Sometimes, signal components might not live the entire duration of the signal. In such cases, the IF estimation is considered valid only where the component is defined, i.e. where it is alive. The remaining time can be filled with zeros.
The first method is based on the classical analytical signal. The latter is a complex signal, the real part of which is the original, real-valued signal, whereas its imaginary part is the Hilbert transform of the original signal. This way, one can estimate the instantaneous frequency as the derivative of the phase of the analytical signal.
[3]:
def method_1(signal):
"""
Basic method for instantaneous frequency estimation
using analytic signal.
"""
# 1. Analytic signal using Hilbert transform
analytic_signal = sg.hilbert(signal)
# 2. Get the derivative of the phase
phase = np.unwrap(np.angle(analytic_signal))
instf = np.diff(phase,append=phase[-1])/2/pi
return [instf,] # List of estimated IFs, even if its only one.
# Check results using one of the signals defined before.
instf = method_1(signal_2)
plt.plot(instf[0], label='Estimated IF')
plt.plot(signal_2.instf[0],'r--', label='Original IF') # signal_2 is a Signal object with info about its IFs.
plt.xlabel('time'); plt.ylabel('norm. frequency')
plt.legend()
[3]:
<matplotlib.legend.Legend at 0x7167dc9f7890>
Notice that the estimation is only valid where the original IF is different from zero (this is taken into account in the benchmarking process).
The second method is based on the spectrogram. As shown before, the instantenous frequency of a signal component defines a trajectory in the time-frequency plane that one can track. We call this trajectory ridge. The method then tracks the ridge in the spectrogram and normalizes it to obtain an estimation of the instantaneous frequency.
[4]:
def get_ridge(stft):
"""
Get a ridge from the stft of a signal
"""
spectrogram = np.abs(stft)**2
ridge = np.zeros((spectrogram.shape[1]))
for i in range(spectrogram.shape[1]):
ridge[i] = np.argmax(spectrogram[:,i])
return ridge
def method_2(signal):
# 1. Compute STFT
stft = get_stft(signal)
# 2. Get ridge from spectrogram
ridge = get_ridge(stft)
return [ridge/N/2,] # List of estimated IFs, even if its only one.
# Check results using one of the signals defined before.
instf = method_2(signal_2)
plt.plot(instf[0], label='Estimated IF')
plt.plot(signal_2.instf[0],'r--', label='Original IF') # signal_2 is a Signal object with info about its IFs.
plt.xlabel('time'); plt.ylabel('norm. frequency')
plt.legend()
[4]:
<matplotlib.legend.Legend at 0x7167dc75f890>
Finally, we create a dictionary of methods to compare.
[5]:
methods = {'method_1':method_1,'method_2':method_2}
Instantiating a benchmark object
[6]:
benchmark = Benchmark(task='inst_frequency',
methods=methods,
# parameters=parameters,
signal_ids=signals,
SNRin=[0,10,20],
repetitions=30,
N = N)
benchmark.run()
Running benchmark...
- Signal linear_chirp
100%|██████████| 3/3 [00:00<00:00, 36.77it/s]
- Signal cos_chirp
100%|██████████| 3/3 [00:00<00:00, 62.48it/s]
- Signal exp_chirp
100%|██████████| 3/3 [00:00<00:00, 62.03it/s]
The test has finished.
[6]:
{'perf_metric': {'linear_chirp': {0: {'method_1': {'((), {})': {'Comp.0': [np.float64(0.023348895819600257),
np.float64(0.03979174079021587),
np.float64(0.06607565605728193),
np.float64(0.04338934228611128),
np.float64(0.044232638890761955),
np.float64(0.041296690561244485),
np.float64(0.0604138964214883),
np.float64(0.03904943041841814),
np.float64(0.045831869768106874),
np.float64(0.035102932761945815),
np.float64(0.03719034671939433),
np.float64(0.03767039093357804),
np.float64(0.034813859660183216),
np.float64(0.06353464350871478),
np.float64(0.03208352311544564),
np.float64(0.06018851665366399),
np.float64(0.0354311415464003),
np.float64(0.034265947390530495),
np.float64(0.03972605695536964),
np.float64(0.028790211978313822),
np.float64(0.03960982624070145),
np.float64(0.026457775711775607),
np.float64(0.04282145109379904),
np.float64(0.04023072928762455),
np.float64(0.04117138176698881),
np.float64(0.03827147135419383),
np.float64(0.04233926157083375),
np.float64(0.06019985816335912),
np.float64(0.037546421668704255),
np.float64(0.03733748953870723)]}},
'method_2': {'((), {})': {'Comp.0': [np.float64(0.0003996497594576028),
np.float64(0.0005240534879169077),
np.float64(0.01050292921816594),
np.float64(0.002819988342263428),
np.float64(0.0002914080708423259),
np.float64(0.0008335739771298727),
np.float64(0.0038863830110109246),
np.float64(0.004043109298168276),
np.float64(0.012912149255358059),
np.float64(0.00031682403119020325),
np.float64(0.0004971461968644312),
np.float64(0.002652664883139295),
np.float64(0.002009163995228785),
np.float64(0.001811552084260413),
np.float64(0.0015354403578967714),
np.float64(0.010096365546569062),
np.float64(0.00020614864991711908),
np.float64(0.0005948440568054132),
np.float64(0.005180716827816281),
np.float64(0.002429183539515842),
np.float64(0.0009681918053577784),
np.float64(0.0019484145678163379),
np.float64(0.005358849962939993),
np.float64(0.013452078002625488),
np.float64(0.011429236904536757),
np.float64(0.003740940622809774),
np.float64(0.0011349895451492416),
np.float64(0.008870681505213432),
np.float64(0.008524747320102517),
np.float64(0.0020501094808609045)]}}},
10: {'method_1': {'((), {})': {'Comp.0': [np.float64(0.01617519625514762),
np.float64(0.017530834336494824),
np.float64(0.013873121025497445),
np.float64(0.019671437535354348),
np.float64(0.012001401230689362),
np.float64(0.009508156045883027),
np.float64(0.01609817665309385),
np.float64(0.013081665889286226),
np.float64(0.01236801595883471),
np.float64(0.007948016843391874),
np.float64(0.018077550527715407),
np.float64(0.013707447464055002),
np.float64(0.011422855482888822),
np.float64(0.02056209041483316),
np.float64(0.005622824775479343),
np.float64(0.012259649742845054),
np.float64(0.01162661033358967),
np.float64(0.01614455991623769),
np.float64(0.012580381274601536),
np.float64(0.014606251623267758),
np.float64(0.009299146075949434),
np.float64(0.005205874573343609),
np.float64(0.010588544675526838),
np.float64(0.00664170228651748),
np.float64(0.02343738337235709),
np.float64(0.005877446522622843),
np.float64(0.009575406375487299),
np.float64(0.03174232375837536),
np.float64(0.022777763106452385),
np.float64(0.020607387849783505)]}},
'method_2': {'((), {})': {'Comp.0': [np.float64(0.00012677350477121522),
np.float64(0.0003974560572949114),
np.float64(0.00021196654534670918),
np.float64(0.0001956752213518489),
np.float64(0.0002028596176630918),
np.float64(0.00029668590617629707),
np.float64(8.656895855815008e-05),
np.float64(0.0005331865983571222),
np.float64(8.796533602928083e-05),
np.float64(8.50074450288136e-05),
np.float64(0.00018874483037224096),
np.float64(0.00025380376569314033),
np.float64(0.0003367505213084616),
np.float64(0.000134952085338143),
np.float64(3.406177600875531e-05),
np.float64(0.0003333458938136903),
np.float64(0.0001280441832822512),
np.float64(0.00024524906602522013),
np.float64(0.0001527824339386236),
np.float64(2.3433315104630195e-05),
np.float64(0.0001413694681158775),
np.float64(0.000242235332962965),
np.float64(0.00014040516044967066),
np.float64(0.0018349607723639417),
np.float64(0.003638078132648407),
np.float64(0.0001671293894902545),
np.float64(0.0001289248364979196),
np.float64(6.809653488925102e-05),
np.float64(5.7485782654525596e-05),
np.float64(0.00031912431129572363)]}}},
20: {'method_1': {'((), {})': {'Comp.0': [np.float64(0.0007583467392263925),
np.float64(0.008539210245406662),
np.float64(0.003853251284062416),
np.float64(0.016767838765666027),
np.float64(0.01103340800766388),
np.float64(0.00997406875343795),
np.float64(0.007396205414425641),
np.float64(0.004675745819189061),
np.float64(0.002929906142928576),
np.float64(0.008328833604573191),
np.float64(0.008059881207440492),
np.float64(0.014572425348335362),
np.float64(0.004540034263268336),
np.float64(0.014488323810604081),
np.float64(0.005842978677045248),
np.float64(0.01281157148597327),
np.float64(0.007115251966110727),
np.float64(0.01797448079284178),
np.float64(0.013615979554316683),
np.float64(0.01697277938512097),
np.float64(0.0076720492597477236),
np.float64(0.006638932856630152),
np.float64(0.0015395019255031455),
np.float64(0.004472388923176506),
np.float64(0.014634659790176623),
np.float64(0.005188398070397494),
np.float64(0.0068126221581318105),
np.float64(0.004796785866765564),
np.float64(0.01276256790552642),
np.float64(0.009712328632312379)]}},
'method_2': {'((), {})': {'Comp.0': [np.float64(0.00012131097779319815),
np.float64(0.00013432956585556464),
np.float64(0.00010114699594912552),
np.float64(0.00016733928611160548),
np.float64(0.00017515054759116894),
np.float64(0.00010956871763333254),
np.float64(9.985545814652675e-05),
np.float64(7.297782639924124e-05),
np.float64(9.669842609692055e-05),
np.float64(8.808049670145065e-05),
np.float64(0.00010835181698335703),
np.float64(0.00019506975868580895),
np.float64(0.0001072229164696622),
np.float64(0.00012804201043938008),
np.float64(6.91223340087108e-05),
np.float64(0.00014823891085295027),
np.float64(0.00012408417714961946),
np.float64(0.00010334265367040089),
np.float64(0.00013212163157206742),
np.float64(6.886648176063543e-05),
np.float64(0.00011633299477544275),
np.float64(8.661100306770646e-05),
np.float64(0.00013094025690303483),
np.float64(0.00010455108023317901),
np.float64(8.777923203736868e-05),
np.float64(0.00010366054058244699),
np.float64(0.00012972683280165312),
np.float64(8.4837854642722e-05),
np.float64(7.497716776711014e-05),
np.float64(0.0001565015804389907)]}}}},
'cos_chirp': {0: {'method_1': {'((), {})': {'Comp.0': [np.float64(0.06713546025141125),
np.float64(0.04391631450357857),
np.float64(0.03468876374857226),
np.float64(0.02433134444194757),
np.float64(0.034244513353550624),
np.float64(0.04031359791008781),
np.float64(0.046448377767963886),
np.float64(0.033546628922290156),
np.float64(0.017839217778833898),
np.float64(0.03322345482386293),
np.float64(0.030950494076459712),
np.float64(0.026774830307156487),
np.float64(0.04732403700826885),
np.float64(0.03523861570385439),
np.float64(0.04216864969537882),
np.float64(0.04808738147409213),
np.float64(0.03032269985669673),
np.float64(0.02523792839389752),
np.float64(0.0379764573992955),
np.float64(0.0499511857548519),
np.float64(0.054431017056141236),
np.float64(0.03343218219147768),
np.float64(0.0290646410258736),
np.float64(0.03863880899226712),
np.float64(0.04401783360777714),
np.float64(0.041990573564394465),
np.float64(0.04080622963375523),
np.float64(0.054170577176719546),
np.float64(0.03400557010080024),
np.float64(0.03176509547646489)]}},
'method_2': {'((), {})': {'Comp.0': [np.float64(0.005027867769568383),
np.float64(0.0014465473380559522),
np.float64(0.0017514262584388011),
np.float64(0.0003996044632311151),
np.float64(0.00403383373603025),
np.float64(0.0014851167487774863),
np.float64(0.010804697255576926),
np.float64(0.004098727215223186),
np.float64(0.005495062016932799),
np.float64(0.009773000902668577),
np.float64(0.0015958385730739716),
np.float64(0.00459520788145804),
np.float64(0.0007080895951123113),
np.float64(0.006936207396440871),
np.float64(0.00774004354882346),
np.float64(0.006061297751224683),
np.float64(0.0006512582901323843),
np.float64(0.00828641703497028),
np.float64(0.007883162304012887),
np.float64(0.0003511904270658229),
np.float64(0.0053566352531849055),
np.float64(0.0010438976017032055),
np.float64(0.00588102806171002),
np.float64(0.009153934536782891),
np.float64(0.00622136087457728),
np.float64(0.0023427967275629143),
np.float64(0.0030139728732287096),
np.float64(0.000624710535214106),
np.float64(0.0016292612679152285),
np.float64(0.010747295601058837)]}}},
10: {'method_1': {'((), {})': {'Comp.0': [np.float64(0.0074202415510260114),
np.float64(0.004525215786526622),
np.float64(0.013895536128199541),
np.float64(0.009432547290445162),
np.float64(0.009289503994354048),
np.float64(0.004352433783541701),
np.float64(0.005211343515328344),
np.float64(0.009285178106924658),
np.float64(0.0021537044498941677),
np.float64(0.008768057703845982),
np.float64(0.00765822054424142),
np.float64(0.007669649368192998),
np.float64(0.005181811514512253),
np.float64(0.00820580115319737),
np.float64(0.002862736885358519),
np.float64(0.004435128291591239),
np.float64(0.0031874705506861937),
np.float64(0.006786522396407998),
np.float64(0.005069743664899327),
np.float64(0.004081715516511666),
np.float64(0.006177326707871613),
np.float64(0.0030966206220662566),
np.float64(0.006797983320791352),
np.float64(0.009914970952949412),
np.float64(0.006784652241574983),
np.float64(0.0062638822440971315),
np.float64(0.0047210295793866895),
np.float64(0.007833245923834133),
np.float64(0.007971682879042739),
np.float64(0.009236788744341789)]}},
'method_2': {'((), {})': {'Comp.0': [np.float64(0.0002780533816757823),
np.float64(7.85804918914274e-05),
np.float64(4.632409561253925e-05),
np.float64(0.00012297888172552055),
np.float64(0.0002487323085125485),
np.float64(0.00011715598939896549),
np.float64(8.06754487385047e-05),
np.float64(0.0007632664381073627),
np.float64(0.0001009173232821352),
np.float64(8.923153313607313e-05),
np.float64(6.997794319971299e-05),
np.float64(0.00015610992308380834),
np.float64(8.516634312207476e-05),
np.float64(8.970259288204324e-05),
np.float64(7.194757482587793e-05),
np.float64(0.0001352020324270963),
np.float64(0.00015369506718292846),
np.float64(8.975699335306731e-05),
np.float64(8.248119846745578e-05),
np.float64(0.000119791036978783),
np.float64(8.909887002266526e-05),
np.float64(9.698598236370811e-05),
np.float64(0.00014051018392516746),
np.float64(0.001274905690751098),
np.float64(9.257238625131505e-05),
np.float64(0.0001464802467516096),
np.float64(8.88420083908612e-05),
np.float64(5.4318759002906455e-05),
np.float64(9.806750705504462e-05),
np.float64(0.00021638391002495966)]}}},
20: {'method_1': {'((), {})': {'Comp.0': [np.float64(0.0030163400182413418),
np.float64(0.001741428770945631),
np.float64(0.006037251540039526),
np.float64(0.007748066213013297),
np.float64(0.0033339308701335934),
np.float64(0.005293510568639822),
np.float64(0.002109897883029059),
np.float64(0.005907912802761008),
np.float64(0.0004746677258947496),
np.float64(0.007663240820494584),
np.float64(0.00405049069438651),
np.float64(0.0050679022667702335),
np.float64(0.004114146136005769),
np.float64(0.005929610962515874),
np.float64(0.0011763291144998012),
np.float64(0.0019044322413731085),
np.float64(0.002327029837895789),
np.float64(0.006051220191460699),
np.float64(0.0018897754786871227),
np.float64(0.0010777610529745746),
np.float64(0.002530477654547212),
np.float64(0.0017556152858099697),
np.float64(0.001959355418750472),
np.float64(0.005476516546844996),
np.float64(0.005326217513764403),
np.float64(0.0017391729968647778),
np.float64(0.0032902154800630292),
np.float64(0.0011606978875274264),
np.float64(0.013328465045554837),
np.float64(0.005222277490023573)]}},
'method_2': {'((), {})': {'Comp.0': [np.float64(5.010116368700873e-05),
np.float64(5.70327912532897e-05),
np.float64(4.251960280754958e-05),
np.float64(5.5514287269228476e-05),
np.float64(0.00011160627220126949),
np.float64(7.32431470143492e-05),
np.float64(4.7639073719906206e-05),
np.float64(7.278405381336114e-05),
np.float64(4.864391447996169e-05),
np.float64(4.956723133507284e-05),
np.float64(5.7806937132616796e-05),
np.float64(6.966585690215086e-05),
np.float64(5.217840420614338e-05),
np.float64(5.3838086585830514e-05),
np.float64(5.575796849985641e-05),
np.float64(7.107235002913263e-05),
np.float64(6.991854417549201e-05),
np.float64(5.5547133924886e-05),
np.float64(5.302922328494471e-05),
np.float64(7.579029911387984e-05),
np.float64(5.164709173092462e-05),
np.float64(7.06958209319083e-05),
np.float64(6.1010322067964156e-05),
np.float64(7.277546616197334e-05),
np.float64(4.682646649496672e-05),
np.float64(7.713166275010349e-05),
np.float64(4.50429683489418e-05),
np.float64(4.491735593891848e-05),
np.float64(5.377537580254456e-05),
np.float64(5.909828798783119e-05)]}}}},
'exp_chirp': {0: {'method_1': {'((), {})': {'Comp.0': [np.float64(0.03677580750530742),
np.float64(0.03551565319391838),
np.float64(0.03676784068470964),
np.float64(0.030701406376701913),
np.float64(0.03253368603212754),
np.float64(0.029916200888793147),
np.float64(0.02722272520624418),
np.float64(0.023953038706149402),
np.float64(0.025349028560320678),
np.float64(0.025867332847503557),
np.float64(0.03138665675665843),
np.float64(0.018992139775609432),
np.float64(0.018581004987902064),
np.float64(0.04145923858543026),
np.float64(0.03714011525906623),
np.float64(0.03296293029955267),
np.float64(0.02380787017873268),
np.float64(0.043695138933205466),
np.float64(0.023399361034023982),
np.float64(0.03649986301621358),
np.float64(0.04029163782767058),
np.float64(0.035940357926863474),
np.float64(0.02765129547867609),
np.float64(0.04137134149337557),
np.float64(0.021544263725525054),
np.float64(0.025346274561539237),
np.float64(0.03266521402848918),
np.float64(0.049532690352138),
np.float64(0.042874202420869506),
np.float64(0.02838847375462569)]}},
'method_2': {'((), {})': {'Comp.0': [np.float64(0.01842704218939824),
np.float64(0.0029459126484588432),
np.float64(0.005277269339811568),
np.float64(0.005519695231673428),
np.float64(0.0023922549846587845),
np.float64(0.0017162335913330708),
np.float64(0.0011285244727690523),
np.float64(0.002916179018172669),
np.float64(0.0021167839473422907),
np.float64(0.002154133261873915),
np.float64(0.011892918990011736),
np.float64(0.0003313331421848894),
np.float64(0.001596461824617257),
np.float64(0.0017852919381354789),
np.float64(0.004840968909552708),
np.float64(0.00391030258819797),
np.float64(0.018877793931226242),
np.float64(0.005642124488442879),
np.float64(0.00833496874315418),
np.float64(0.007890210777038782),
np.float64(0.0024796331726072915),
np.float64(0.0020669210996269055),
np.float64(0.0005639071070792048),
np.float64(0.007751693240202398),
np.float64(0.007040742720055547),
np.float64(0.0062865814331663),
np.float64(0.0013423481449511155),
np.float64(0.0057983265818411555),
np.float64(0.003087828788231774),
np.float64(0.0012369215734523216)]}}},
10: {'method_1': {'((), {})': {'Comp.0': [np.float64(0.010738786474278658),
np.float64(0.013802882027159756),
np.float64(0.01687042036494829),
np.float64(0.013262290330917297),
np.float64(0.013519126714961518),
np.float64(0.009262827734237453),
np.float64(0.009399266182708707),
np.float64(0.0058635249994053494),
np.float64(0.00420567976054402),
np.float64(0.00719588375655173),
np.float64(0.012463014447718227),
np.float64(0.007769467950306802),
np.float64(0.004648749837753013),
np.float64(0.02202996233586865),
np.float64(0.0053660396704958285),
np.float64(0.01199584818740554),
np.float64(0.005864695128497068),
np.float64(0.011391848489456456),
np.float64(0.012700539807646828),
np.float64(0.011639859272662444),
np.float64(0.013932483768640189),
np.float64(0.005733722430154064),
np.float64(0.0027172090182651675),
np.float64(0.015637260235499145),
np.float64(0.011344378736426491),
np.float64(0.005567435266007055),
np.float64(0.007413559487125304),
np.float64(0.016284979796288145),
np.float64(0.009238342348092085),
np.float64(0.014442786782072023)]}},
'method_2': {'((), {})': {'Comp.0': [np.float64(0.00035643612725529784),
np.float64(0.0003079207999692433),
np.float64(0.00037572569733007165),
np.float64(0.0002816251811158664),
np.float64(0.0002704646311960085),
np.float64(0.0001954278777493699),
np.float64(0.00014037353661370297),
np.float64(0.0008247429876478514),
np.float64(0.0003044601009147114),
np.float64(0.00021793572568970288),
np.float64(0.00015435785236527186),
np.float64(0.0002402096356212838),
np.float64(0.0001861572393418597),
np.float64(0.0004933956332376928),
np.float64(0.0001543579157089018),
np.float64(0.0006321418363534974),
np.float64(0.00017979717572875326),
np.float64(0.0001559465510919802),
np.float64(0.00044669600947488784),
np.float64(0.0002142042870854082),
np.float64(0.00042796224850322656),
np.float64(0.0004935509630436006),
np.float64(0.00047796647426810376),
np.float64(0.001672382887461569),
np.float64(0.0010825764190505194),
np.float64(0.0003751312071261922),
np.float64(0.00026307181662476355),
np.float64(0.0002028364554471725),
np.float64(0.00010634285222379932),
np.float64(0.0005331404978170177)]}}},
20: {'method_1': {'((), {})': {'Comp.0': [np.float64(0.0007237827326920581),
np.float64(0.008789312206833832),
np.float64(0.007141993886385917),
np.float64(0.008079139772341051),
np.float64(0.012119399712150184),
np.float64(0.002336786349071315),
np.float64(0.00821945697139762),
np.float64(0.004246035539334252),
np.float64(0.00392506752232883),
np.float64(0.008223484339565393),
np.float64(0.008170768827501725),
np.float64(0.005511721073265225),
np.float64(0.004354847338329765),
np.float64(0.015237560624833649),
np.float64(0.0047994829211792265),
np.float64(0.013186658534998686),
np.float64(0.0010564258455054265),
np.float64(0.008159341857642355),
np.float64(0.0072265855910970695),
np.float64(0.010049713382782302),
np.float64(0.007201180241581152),
np.float64(0.008497319560221511),
np.float64(0.0016396451212835012),
np.float64(0.00875633829080253),
np.float64(0.006255304949046061),
np.float64(0.004859169136081111),
np.float64(0.006748572972958572),
np.float64(0.006483747247248712),
np.float64(0.006982635532762834),
np.float64(0.013377054572411824)]}},
'method_2': {'((), {})': {'Comp.0': [np.float64(0.00032493261574456823),
np.float64(0.000289124553879715),
np.float64(0.0003422792030885124),
np.float64(0.0002780611952529101),
np.float64(0.0003080283201682128),
np.float64(0.0002177946718286901),
np.float64(0.0001900891455858367),
np.float64(0.000292202488724123),
np.float64(0.00021171854353544035),
np.float64(0.00038775350097304344),
np.float64(0.00021691946168140376),
np.float64(0.0002717702074134243),
np.float64(0.0003837928287335054),
np.float64(0.0003312647507645092),
np.float64(0.0002080999376327204),
np.float64(0.00028188833321330666),
np.float64(0.00021878966129653622),
np.float64(0.000200256357008888),
np.float64(0.0003866887720471437),
np.float64(0.00023990605672819372),
np.float64(0.000350349308344533),
np.float64(0.0003819011006422336),
np.float64(0.00040115776630415633),
np.float64(0.00022324893292307237),
np.float64(0.00015633897869330227),
np.float64(0.0003506555888282646),
np.float64(0.00027582497032353526),
np.float64(0.00022230792575164628),
np.float64(0.0002049335153061638),
np.float64(0.0004211929407041127)]}}}}}}
Displaying results.
[7]:
results = benchmark.results # Get dictionary with the results.
df = benchmark.dic2df(results) # Transform dictionary to DataFrame
Before displaying the results, we need to format the DataFrame in the correct way:
[8]:
df = df.reset_index()
df = df.iloc[:,[3,4,1,5,2,6]]
col_names = list(df.columns)
col_names[0:6] = ['Method','Parameter', 'Signal_id','Repetition','SNR','QRF']
df.columns = col_names
df = df.pivot_table(index= ['Method','Parameter', 'Signal_id','Repetition'], columns='SNR', values='QRF')
df = df.reset_index()
df
[8]:
| SNR | Method | Parameter | Signal_id | Repetition | 0 | 10 | 20 |
|---|---|---|---|---|---|---|---|
| 0 | method_1 | ((), {}) | cos_chirp | 0 | 0.067135 | 0.007420 | 0.003016 |
| 1 | method_1 | ((), {}) | cos_chirp | 1 | 0.043916 | 0.004525 | 0.001741 |
| 2 | method_1 | ((), {}) | cos_chirp | 2 | 0.034689 | 0.013896 | 0.006037 |
| 3 | method_1 | ((), {}) | cos_chirp | 3 | 0.024331 | 0.009433 | 0.007748 |
| 4 | method_1 | ((), {}) | cos_chirp | 4 | 0.034245 | 0.009290 | 0.003334 |
| ... | ... | ... | ... | ... | ... | ... | ... |
| 175 | method_2 | ((), {}) | linear_chirp | 25 | 0.003741 | 0.000167 | 0.000104 |
| 176 | method_2 | ((), {}) | linear_chirp | 26 | 0.001135 | 0.000129 | 0.000130 |
| 177 | method_2 | ((), {}) | linear_chirp | 27 | 0.008871 | 0.000068 | 0.000085 |
| 178 | method_2 | ((), {}) | linear_chirp | 28 | 0.008525 | 0.000057 | 0.000075 |
| 179 | method_2 | ((), {}) | linear_chirp | 29 | 0.002050 | 0.000319 | 0.000157 |
180 rows × 7 columns
Finally, we can use the functionality from the ResultsInterpreter class to display the result on interactive plots (using plotly).
Here we display the mean squared error (MSE), which is the default performance metric for instantaneous frequency estimation (the lower, the better).
[9]:
# Summary interactive plots with Plotly and a report.
from plotly.offline import iplot
import plotly.io as pio
pio.renderers.default = "plotly_mimetype+notebook"
interpreter = ResultsInterpreter(benchmark)
figs = interpreter.get_summary_plotlys(df, bars=True, ylabel='Mean Squared Error')
for fig in figs:
iplot(fig)