**COMPONENT ISOLATION AND RECONSTRUCTION (FILTERING) :**
This tutorial covers the signal filtering capabilities of AutoSignal. Signal components can be isolated or removed from data using Fourier, Eigen, and Wavelet decomposition and reconstruction techniques.
**Generating A Test Signal**
Select the Generate Signal option in the Edit menu or Main toolbar.
For this tutorial, we will create a data stream consisting of three sinusoids and white noise. The sinusoids will vary from one another in both power and frequency in order to illustrate the benefits of both Fourier filtering and Eigenfiltering.
Click Read and select the file tutor7.sig from the Signals subdirectory.
The following signal expression is imported:
SRATE=5000
NYQ=SRATE/2
AMP1=1
AMP2=SQRT(2)/2
AMP3=0.5
FREQ1=NYQ*0.1
FREQ2=NYQ*0.2
FREQ3=NYQ*0.3
PHASE1=PI/2
PHASE2=PI
PHASE3=3*PI/2
F1=AMP1*SIN(2*PI*X*FREQ1+PHASE1)
F2=AMP2*SIN(2*PI*X*FREQ2+PHASE2)
F3=AMP3*SIN(2*PI*X*FREQ3+PHASE3)
Y=F1+F2+F3
The X (time) values vary from 0 to 0.1023 with a 0.0002 sample increment. The Nyquist frequency is 2500 (half the sampling rate). The first spectral peak is at frequency 250. The second peak is at frequency 500 and contains half the power of the first. The final peak is at frequency 750 and contains one-quarter the power of the initial peak. Gaussian noise is added at a 10% level. The aim is to explore the isolation and recovery of each of the individual sinusoids.
Click OK to process the current signal.
An AutoSignal graph is presented containing the 512 point generated data.
Click OK to accept the generated data. Click Yes when asked to update the main data table with the revised data.
**Fourier Domain Filtering**
Select the Fourier Filtering and Reconstruction option in the Process menu and toolbar. Select the cs3 BHarris 3 window. Be sure the edge% is 1.0, the transform is Best Exact N, Nmin is set to 512, and the plot is set to dB.
Be sure the XY Sectioning Mode button is selected in the upper graph's toolbar.
This option is the main frequency domain filtering procedure within AutoSignal. It can be used to include or exclude any region of frequency-magnitude space within the Fourier decomposition. It is a simple matter to construct low-pass, high-pass, band-pass, and band-stop filters. You can easily generate multiple band-pass and band-stop zones even to the point of toggling individual frequencies on and off. The lower graph displays the reconstructed data. The reconstruction using this option is via the inverse FFT, so interpolation is not available.
AutoSignal offers the full selection of data tapering windows in this Fourier domain filtration procedure. This makes it possible to reconstruct components containing very low power that could not otherwise be isolated using Fourier techniques. The drawback to data windowing is that the information nearer the bounds of the data stream is more susceptible to noise in the reconstruction. The edge% specifies a window threshold. Typically, it is best to avoid reconstruction in areas where the data taper drops to less than 1% of its peak value. For this reason, the reconstruction using most of tapering windows will not include some number of points at the two edges of the data stream.
Two exceptions are the Hamming or "raised cosine" window (cs2 Hamming) and the Blackman-Harris 3-parameter (cs3 BHarris 3) window. These windows do not decay to zero at the bounds, enabling the full reconstruction of the data.
Using the left mouse button, box the four highest bars associated with the second peak. Box only the four peak positions.
This isolates the second component.
Click OK to close the Fourier filtration procedure and answer Yes to update the main data table.
Select the Fourier Spectrum option from the Spectral menu or toolbar. Use the Best Exact N algorithm with Nmin set to the data length 512. Change the plot to dB Norm, and set the signal count in the sig field to 1.
When the data are Fourier filtered using a window and a subsequent non-window Fourier analysis follows, the filtered peak displays a softened transition.
Select the Section Local Copy of Data option.
Be sure the X Sectioning Mode is selected.
At the left edge of the graph, click and hold the left mouse button down while sliding the mouse right toggling off the data elements showing edge effects on the left side of the data stream. Click again where the right edge effects first begin to appear and hold the left mouse button down while sliding to the right edge of the graph.
Click OK to accept the sectioning.
Click the Non-Linear Optimization button. Accept the default settings and click the OK button. At the conclusion of the iterative fit, click the Review Fit button.
Click the Numeric Summary button in the Non-Linear Optimization Review. Inspect the fit results.
**Fitted Parameters**
**r² Coef Det** |
**DF Adj r²** |
**Fit Std Err** |
**F-value** |
0.99842724 |
0.99841517 |
0.02001047 |
1.2443e+059 |
**Data Power** |
**Model Power** |
**Error Power** |
0.0199603715 |
0.0199313575 |
3.139283e-05 |
**Comp** |
**Type** |
**Frequency** |
**Amplitude** |
**Phase** |
1 |
Sine |
499.907005 |
0.71028897 |
3.17363736 |
The parameters of this second component are recovered with a high degree of accuracy. The goodness of fit statistics suggest that the sinusoid was almost perfectly reconstructed.
Close the Numeric Summary and close the Non-Linear Optimization Review.
Close the Fourier spectrum window with the OK button.
Select the Fourier Spectrum with Data Window option from the Spectral menu or toolbar. Select the cs3 BHarris 3 window. Use the Best Exact N algorithm with Nmin set to the data length 512. Change the plot to dB Norm, and set the signal count in the sig field to 1.
When the same window is used for the subsequent analysis as that used in the filtration, the result is a true "brickwall" filter. Only the four frequencies retained appear in the spectrum and all others are at about -320dB (the limit of machine precision). The same brickwall is present if no window is used in both instances.
Change Nmin to 1024.
The brickwall filter response disappears when zero padding is used. The point to bear in mind is that Fourier filtration always produces true brickwall frequency transitions, but only in the mode in which the filtration was made. If a filtration is made without a window and without zero padding, any subsequent analysis must be made without a window and without zero padding in order to see this brickwall transition. A brickwall frequency response thus has as much to do with how the subsequent frequency analysis of the data is done.
Since a data tapering window minimizes spectral leakage, the most accurate reconstructions will probably involve the use of data tapering windows. If you are seeking to isolate and reconstruct very low power components, a data tapering window will likely be mandatory. When the greater susceptibility to reconstruction error at the bounds results in edge effects, you can section the reconstruction as in the above example, or you can set an edge% at a level where the two boundary regions are automatically removed from the reconstruction.
Close the Fourier spectrum window with the OK button.
Click the Reset XY Data button in the main toolbar to restore the original unfiltered data.
Once again, select the Fourier Filtering and Reconstruction option.
Using the right mouse button, box the four highest bars associated with the second peak. Box only the four peak positions.
This removes the second component. Analyzing the data in a manner analogous to the previous example, the following fit results are attained:
**Fitted Parameters **
**r² Coef Det** |
**DF Adj r²** |
**Fit Std Err** |
**F-value** |
0.98427518 |
0.98406782 |
0.09954069 |
5708.54936 |
**Data Power** |
**Model Power** |
**Error Power** |
0.0574659413 |
0.0565723757 |
0.0009036414 |
**Comp** |
**Type** |
**Frequency** |
**Amplitude** |
**Phase** |
1 |
Sine |
250.016468 |
0.99029669 |
1.56163104 |
2 |
Sine |
750.053700 |
0.49623799 |
4.70220567 |
The error is somewhat higher since the noise was not removed with this filtration. Note the near complete recovery of the first and third components.
Using the left mouse button, box the four highest bars on all three of the peaks.
This filters out virtually all of the noise in the signal.
Using the right mouse button, box the four highest bars on all three of the peaks.
This filters out the sinusoids leaving only the background.
Enable the X Sectioning Mode.
Click the Reset button to restore an unfiltered state.
In this mode frequency bins are toggled off when the mouse is held down and moved to the right in the graph. Bins are toggled back on by holding the mouse down and moving to the left. To toggle an individual bin on and off left click the point (or top of the bar). Note that bar plots are used by default for data sizes of 512 or less. Two different point format configurations are saved across sessions, one for data sets with 513 or more points and another for smaller sets.
Experiment with turning frequency bands and individual bins on and off. Note that this approach enables you to create any frequency domain filter configuration desired.
Click OK to close the Fourier filtration procedure and answer No to update the main data table.
**Eigendecomposition Filtration**
There are two key ways to isolate spectral components. The Fourier approach isolates components by frequency. The Eigendecomposition method isolates spectral components by signal strength.
Select the Eigendecomposition Filtering and Reconstruction option in the Process menu and toolbar. Be sure the CovM FB algorithm is selected, that a matrix order of 50 is set, that the Eigenvalues Normalize % box is checked, and that FFT is checked for the lower graph. Using the left mouse button, box the first two eigenmodes in the upper graph.
This procedure offers an FFT rendering using a Kaiser-Bessel window whose width is automatically determined based on the length of the data record. This makes it easy to get an immediate frequency domain representation of the content of one or more eigenmodes. Note that this is a convenience for immediately viewing non-optimized representations of component power spectra.
Oscillatory components appear as pairs in an eigendecomposition. The magnitude of the eigenvalues is related to signal strength. The first two eigenmodes are associated with the first (amplitude 100) component, the second two with the half power (amplitude 70.71) component, and the final pair with the quarter power (amplitude 50) component. Note that the first two eigenmodes completely recover the first component, but a small amount of the second component is also present. The separation is not complete.
Right click the Order entry field and select 100 from the popup menu. Again enclose the first two eigenmodes using the left mouse button.
The first component is now fully isolated within the dynamic range of the FFT.
Enclose the third and fourth eigenmodes.
The second component is now fully isolated within the dynamic range of the FFT.
Enclose the fifth and sixth eigenmodes.
Within the dynamic range of the FFT, the lowest power component is completely isolated and reconstructed.
Enclose the first six eigenmodes using the left mouse button.
The signal components are recovered and the background removed. The Eigendecomposition Smoothing and Denoising option is based upon this type of eigenmode thresholding.
Enclose the first six eigenmodes using the right mouse button.
The reverse eigenfiltration has occurred. The signal components are removed and the background is retained. This is useful for separating backgrounds for subsequent modeling.
Again enclose the first six eigenmodes using the left mouse button. Select FFT Comp. for the lower graph.
When dealing with data where oscillatory pairs are not present or obvious, this option is useful for displaying a Fourier spectrum of all selected eigenmodes.
Select AR Comp. for the lower graph.
The AR options generate a basic autoregressive spectrum using the Data FB (modified covariance) AR algorithm. The model order is usually 40, but is lower if necessary to produce a successful AR fit. This makes it easy to get an immediate frequency spectrum of the content of one or more eigenmodes. As with the FFT options, the AR options are a convenience for immediately viewing non-optimized AR representations of the components. Keep in mind that AR spectral peaks are not linearly related to the power of components. Also this AR representation was chosen for speed. Lacking the SVD noise removal, these AR spectra can readily be judged as far from optimum as peaks arising from noise will be present.
In this example, the AR spectra indicate that the eigenmode separation was not as complete as the FFT had suggested. AR spectral peaks are proportional to the square of the power of components, and as such trace components are greatly emphasized. This is also why peaks that arise from noise are so evident in AR spectra. We will now quantify the separation.
Select the AR option. Use the left mouse button to enclose the fifth and sixth eigenmodes.
Clearly there is a trace of the 250 and 500 frequency components in the 750 frequency component.
Click OK to close the Eigenfiltration procedure and answer Yes to update the main data table.
Select the AR (AutoRegressive) Spectrum option in the Spectral menu or toolbar. To match the AR spectrum in the Eigenfiltering procedure, set the algorithm to Data FB and the model order to 40. Be sure Full Range and Adaptive n are checked and that the plot format is dB.
Clearly the AR spectrum picked up traces of the other two components. Again, keep in mind that this is not an optimized AR spectrum. This algorithm and model order are used in the Eigenfiltering procedure for performance reasons.
Click the Numeric Summary button.
**AR Fit Details**
**r²** |
**StdErr** |
**F-stat** |
**SSE/n** |
0.9999996822 |
0.0002066806 |
3.712704e+07 |
3.937963e-08 |
**Sine Component Fit (Suboptimal)**
**Frequency** |
**Amplitude** |
**Phase** |
**Power ** |
**%** |
**Rel %** |
250.360109 |
0.00022544 |
3.10806969 |
2.5897e-09 |
2.059e-05 |
2.0596e-05 |
518.153112 |
0.00086478 |
3.30540348 |
3.8247e-08 |
0.00030410 |
0.00030419 |
524.551528 |
0.00090036 |
4.43235519 |
4.1501e-08 |
0.00032997 |
0.00033007 |
750.172971 |
0.49578157 |
4.65920895 |
0.01257358 |
99.9710261 |
100.000000 |
763.602327 |
0.00831703 |
2.90257453 |
3.533e-06 |
0.02809033 |
0.02809847 |
**r²** |
**DOF r²** |
**Std Err** |
**F-stat** |
0.9989394459 |
0.9988062927 |
0.0121600608 |
7652.9649363 |
When SVD is not used to specify a signal space, all of the roots in the AR model are assigned as spectral peaks. Only the pertinent roots are shown above. Two roots occurred near the 750 primary frequency. They contain 99.998% of the power in the signal. The two roots near 500 contain only 0.0006% of the power. The root at 250 contains only 0.00002% of the power. Despite the suggestion of incomplete eigenfiltering in the AR spectral plot, from a practical quantitative perspective, the separation can be considered complete. Note also the exceedingly high goodness of fit values for both the AR model fit and the linear sinusoidal fit. The eigenfiltration removed virtually all of the noise.
Close the AR Numeric Summary. Close the AR spectral procedure.
Importing an SST (Sea Surface Temperature) Anomaly Data Set
Select the Import option in the File menu or main toolbar. Change the format to Excel (xls) and select sample.xls from the Signals subdirectory. Check the Import Preview option. Click on (9)Wavelet!G: Year. This column in the Excel worksheet is used as the X or time variable. Click on (9)Wavelet!I: Anomaly(C). This column is used as the Y or signal variable.
Click OK to accept the data and OK once again to accept the default titles.
This data set contains the data used for the analysis of El Nino events by:
- Christopher Torrence and Gilbert Compo, "A Practical Guide to Wavelet Analysis", Bulletin of the American Meteorological Society, v.79, no.1, p.61-78. January 1998
For those who want to understand the subtleties of applying wavelet analysis to real-world data, this paper is an excellent reference. It can be found on the Internet at http://paos.colorado.edu/research/wavelets/. It is also included in the PDF directory of the AutoSignal distribution disk.
We will use this data set to explore the reconstruction of components in a non-stationary data series using wavelet filtering.
**Wavelet Filtering**
Select the Wavelet Filtering and Reconstruction option in the Process menu or toolbar. Be sure the Gauss Deriv wavelet is selected, that Complex is checked, and that the wavelet adjustable parameter is 20. Select the dB Norm plot and set the dB lim value to 12.
Be sure the Logarithmic Y Axis is enabled in the upper graph.
Enable Zoom-in Mode in the upper graph. Zoom in the band from 0.1 frequency to 1.0 frequency across all times.
Toggle off the Logarithmic Y Axis in the upper graph.
Clearly the El-Nino Southern Oscillation data are non-stationary. Note that a great deal of power exists within this frequency range. Note also the wavelength or period shown on the right Y axis. Much of the power is located in bands that have a 2-4 year period and in bands that have a 4-8 year period.
Zoom-in once again across all times for frequencies from 0.1 to 0.5.
Select XY Sectioning Mode in the upper graph.
Wavelet filtering can be done by setting thresholds in time, frequency, or spectral magnitude, and also by graphically selecting a region of time-frequency space.
To see the energy in the 2-4 year period, enter 0.25 in the Freq1 field and 0.5 in the Freq2 field.
The reconstruction includes only the oscillations that have this 2-4 year wavelength. The relative calm between 1930 and 1960 correlates with an absence of El Nino events.
To see the energy in the 4-8 year period, use the left mouse to zoom in a rectangle whose boundaries include 0.125 frequency and 0.25 frequency.
Note that the oscillations in this frequency band are more constant across time.
Experiment with isolating time-frequency regions of high energy by enclosing them using graphical sectioning.
Click OK to exit the wavelet filtering procedure. |