Example 2: Frequency-Domain Filtering
+ +Introduction
+ +This example shows how to apply a filter to an audio file using +the Gaborator library, by turning the audio into spectrogram +coefficients, modifying the coefficients, and resynthesizing audio +from them.
+ +The specific filter implemented here is a 3 dB/octave lowpass +filter. This is sometimes called a pinking filter because it +can be used to produce pink noise from white noise. In practice, the +3 dB/octave slope is only applied above some minimum frequency, for +example 20 Hz, because otherwise the gain of the filter would approach +infinity as the frequency approaches 0, and the impulse response would +have to be infinitely wide. +
+ +Since the slope of this filter is not a multiple of 6 dB/octave, it +is difficult to implement as an analog filter, but by filtering +digitally in the frequency domain, arbitrary filter responses such as +this can easily be achieved. +
+ +Preamble
+ +
+#include <memory.h>
+#include <iostream>
+#include <sndfile.h>
+#include <gaborator/gaborator.h>
+
+int main(int argc, char **argv) {
+ if (argc < 3) {
+ std::cerr << "usage: filter input.wav output.wav\n";
+ exit(1);
+ }
+
+
+Reading the Audio
+ +The code for reading the input audio file is identical to +that in Example 1:
+ +
+ SF_INFO sfinfo;
+ memset(&sfinfo, 0, sizeof(sfinfo));
+ SNDFILE *sf_in = sf_open(argv[1], SFM_READ, &sfinfo);
+ if (! sf_in) {
+ std::cerr << "could not open input audio file: "
+ << sf_strerror(sf_in) << "\n";
+ exit(1);
+ }
+ double fs = sfinfo.samplerate;
+ sf_count_t n_frames = sfinfo.frames;
+ sf_count_t n_samples = sfinfo.frames * sfinfo.channels;
+ std::vector<float> audio(n_samples);
+ sf_count_t n_read = sf_readf_float(sf_in, audio.data(), n_frames);
+ if (n_read != n_frames) {
+ std::cerr << "read error\n";
+ exit(1);
+ }
+ sf_close(sf_in);
+
+
+Spectrum Analysis Parameters
+ +The spectrum analysis works much the same as in Example 1, +but uses slightly different parameters. +We use a larger number of frequency bands per octave (100) +to minimize ripple in the frequency response, and the +reference frequency argument is omitted as we don't care about the +exact alignment of the bands with respect to a musical scale.
++ gaborator::parameters params(100, 20.0 / fs); + gaborator::analyzer<float> analyzer(params); ++ +
Precalculating Gains
+ +The filtering will be done by multiplying each spectrogram
+coefficient with a frequency-dependent gain. To avoid having to
+calculate the gain on the fly for each coefficient, which would
+be slow, we will precalculate the gains into a vector band_gains
+of one gain value per band, including one for the
+special lowpass band that contains the frequencies from 0 to 20 Hz.
+ std::vector<float> band_gains(analyzer.bands_end()); ++ +
First, we calculate the gains for the bandpass bands.
+For a 3 dB/octave lowpass filter, the voltage gain needs to be
+proportional to the square root of the inverse of the frequency.
+To get the frequency of each band, we call the
+analyzer method band_ff(), which
+returns the center frequency of the band in units of the
+sampling frequency. The gain is normalized to unity at 20 Hz.
+
+ for (int band = analyzer.bandpass_bands_begin(); band < analyzer.bandpass_bands_end(); band++) {
+ double f_hz = analyzer.band_ff(band) * fs;
+ band_gains[band] = 1.0 / sqrt(f_hz / 20.0);
+ }
+
+
+The gain of the lowpass band is set to the the same value as the +lowest-frequency bandpass band, so that the overall filter gain +plateaus smoothly to a constant value below 20 Hz.
+ ++ band_gains[analyzer.band_lowpass()] = band_gains[analyzer.bandpass_bands_end() - 1]; ++ +
De-interleaving
+ +To handle stereo and other multi-channel audio files,
+we will loop over the channels and filter each channel separately.
+Since libsndfile produces interleaved samples, we first
+de-interleave the current channel into a temporary vector called
+channel:
+ for (sf_count_t ch = 0; ch < sfinfo.channels; ch++) {
+ std::vector<float> channel(n_frames);
+ for (sf_count_t i = 0; i < n_frames; i++)
+ channel[i] = audio[i * sfinfo.channels + ch];
+
+Spectrum Analysis
+Now we can spectrum analyze the current channel, producing +a set of coefficients:
++ gaborator::coefs<float> coefs(analyzer); + analyzer.analyze(channel.data(), 0, channel.size(), coefs); ++ +
Filtering
+
+The filtering is done using the function
+process(), which applies a user-defined function
+to each spectrogram coefficient. Here, that user-defined function is a
+lambda expression that multiplies the coefficient by the appropriate
+precalculated frequency-dependent gain, modifying the coefficient in
+place. The unused int64_t argument is the time in units
+of samples; this could be use to implement a time-varying filter if
+desired.
+The second and third argument to process() specify a
+range of frequency bands to process; here we pass INT_MIN,
+INT_MAX to process all of them. Similarly, the fourth and
+fifth argument specify a time range to process, and we pass
+INT64_MIN, INT64_MAX to process all the coefficients
+in coefs regardless of time.
+
+ process([&](int band, int64_t, std::complex<float> &coef) {
+ coef *= band_gains[band];
+ },
+ INT_MIN, INT_MAX,
+ INT64_MIN, INT64_MAX,
+ coefs);
+
+
+Resynthesis
+We can now resynthesize audio from the filtered coefficients by
+calling synthesize(). This is a mirror image of the call to
+analyze(): now the coefficients are the input, and
+the buffer of samples is the output. The channel
+vector that originally contained the input samples for the channel
+is now reused to hold the output samples.
+ analyzer.synthesize(coefs, 0, channel.size(), channel.data()); ++ +
Re-interleaving
+The audio vector that contained the
+original interleaved audio is reused for the interleaved
+filtered audio. This concludes the loop over the channels.
+
+ for (sf_count_t i = 0; i < n_frames; i++) + audio[i * sfinfo.channels + ch] = channel[i]; + } ++ +
Writing the Audio
+The filtered audio is written using libsndfile,
+using code that closely mirrors that for reading.
+Note that we use SFC_SET_CLIPPING
+to make sure that any samples too loud for the file format
+will saturate; by default, libsndfile makes them
+wrap around, which sounds really bad.
+ SNDFILE *sf_out = sf_open(argv[2], SFM_WRITE, &sfinfo);
+ if (! sf_out) {
+ std::cerr << "could not open output audio file: "
+ << sf_strerror(sf_out) << "\n";
+ exit(1);
+ }
+ sf_command(sf_out, SFC_SET_CLIPPING, NULL, SF_TRUE);
+ sf_count_t n_written = sf_writef_float(sf_out, audio.data(), n_frames);
+ if (n_written != n_frames) {
+ std::cerr << "write error\n";
+ exit(1);
+ }
+ sf_close(sf_out);
+
+
+
+Postamble
++We need a couple more lines of boilerplate to make the example a +complete program: +
++ return 0; +} ++ +
Compiling
+Like Example 1, this example +can be built using a one-line build command: +
++c++ -std=c++11 -I.. -O3 -ffast-math `pkg-config --cflags sndfile` filter.cc `pkg-config --libs sndfile` -o filter ++
Or using the vDSP FFT on macOS:
++c++ -std=c++11 -I.. -O3 -ffast-math -DGABORATOR_USE_VDSP `pkg-config --cflags sndfile` filter.cc `pkg-config --libs sndfile` -framework Accelerate -o filter ++
Or using PFFFT (see Example 1 for how to download and build PFFFT):
++c++ -std=c++11 -I.. -Ipffft -O3 -ffast-math -DGABORATOR_USE_PFFFT `pkg-config --cflags sndfile` filter.cc pffft/pffft.o pffft/fftpack.o `pkg-config --libs sndfile` -o filter ++ +
Running
+Running the following shell commands will download an example +audio file containing five seconds of white noise and filter it, +producing pink noise.
++wget http://download.gaborator.com/audio/white_noise.wav +./filter white_noise.wav pink_noise.wav ++ +
Frequency response
+The following plot shows the actual measured frequency response of the +filter, with the expected 3 dB/octave slope above 20 Hz and minimal +ripple:
+
+
+
+
+
+
diff --git a/lib/gaborator/doc/gen/allkernels_v1_bpo12_ffmin0.03_ffref0.5_anl_wob.png b/lib/gaborator/doc/gen/allkernels_v1_bpo12_ffmin0.03_ffref0.5_anl_wob.png
new file mode 100644
index 0000000..b75fa6c
Binary files /dev/null and b/lib/gaborator/doc/gen/allkernels_v1_bpo12_ffmin0.03_ffref0.5_anl_wob.png differ
diff --git a/lib/gaborator/doc/gen/allkernels_v1_bpo12_ffmin0.03_ffref0.5_syn_wob.png b/lib/gaborator/doc/gen/allkernels_v1_bpo12_ffmin0.03_ffref0.5_syn_wob.png
new file mode 100644
index 0000000..03ff1a1
Binary files /dev/null and b/lib/gaborator/doc/gen/allkernels_v1_bpo12_ffmin0.03_ffref0.5_syn_wob.png differ
diff --git a/lib/gaborator/doc/gen/grid_v1_bpo12_ffmin0.03_ffref0.5_wob.png b/lib/gaborator/doc/gen/grid_v1_bpo12_ffmin0.03_ffref0.5_wob.png
new file mode 100644
index 0000000..91fdf7b
Binary files /dev/null and b/lib/gaborator/doc/gen/grid_v1_bpo12_ffmin0.03_ffref0.5_wob.png differ
diff --git a/lib/gaborator/doc/index.html b/lib/gaborator/doc/index.html
new file mode 100644
index 0000000..d7d3231
--- /dev/null
+++ b/lib/gaborator/doc/index.html
@@ -0,0 +1,78 @@
+
+
+
+
+
+
+The Gaborator
+The Gaborator is a library that generates constant-Q spectrograms +for visualization and analysis of audio signals. It also supports a +fast and accurate inverse transformation of the spectrogram coefficients +back into audio for spectral effects and editing.
+ +The Gaborator implements the invertible constant-Q transform of +Velasco, Holighaus, Dörfler, and Grill, described in the papers + +Constructing an invertible constant-Q transform with nonstationary Gabor frames, 2011 +and +A Framework for invertible, real-time constant-Q transforms, 2012, +using Gaussian bandpass filters and an efficient multi-rate architecture. +
+ +The Gaborator is written in C++11 and compatible with C++14 and C++17. +It has been tested on macOS, Linux, NetBSD, FreeBSD, and iOS, on Intel +x86_64 and ARM processors.
+ +The Gaborator is open source under the GNU Affero General Public +License, version 3, and is also available for commercial licensing. +See the file LICENSE for details.
+ +Example Code
+ +The following examples demonstrate the use of the library in
+various scenarios. They are presented in a "literate
+programming" style, with the code embedded in the commentary
+rather than the other way around.
+Concatenating the code fragments in each example yields a complete C++
+program, which can also be found as a .cc file in
+the examples/ directory.
-
+
- Example 1: Rendering a Spectrogram Image +
- Example 2: Frequency-Domain Filtering +
- Example 3: Streaming +
- Example 4: Measuring the Signal-to-Noise Ratio +
- Example 5: Synthesis from Scratch +
API Reference
+The following documents define the library API. +
+ + +How it Works
+The following document outlines the operation of the library.
+ + +FAQ
+ + +Contact
+Email questions and bug reports to the author at info@gaborator.com.
+ + + diff --git a/lib/gaborator/doc/overview.html b/lib/gaborator/doc/overview.html new file mode 100644 index 0000000..cfb3857 --- /dev/null +++ b/lib/gaborator/doc/overview.html @@ -0,0 +1,135 @@ + + + + + +Overview of Operation
+ +The Gaborator performs three main functions:
+-
+
- spectrum analysis, which turns a signal into a set +of spectrogram coefficients +
- resynthesis (aka reconstruction), which turns a +set of coefficients back into a signal, and +
- rendering, which +turns a set of coefficients into a rectangular array of +amplitude values that can be turned into pixels to display +a spectrogram. +
The following sections give a high-level overview of each +of these functions.
+ +Analysis
+ +The first step of the analysis is to run the signal through +an analysis filter bank, to split it into a number of +overlapping frequency bands.
+ +The filter bank consists of a number of logarithmically spaced +Gaussian bandpass filters and a single lowpass filter. Each bandpass +filter has a bandwidth proportional to its center frequency, which +means they all have the same quality factor Q and form +a constant-Q filter bank. The highest-frequency bandpass +filter will have a center frequency close to half the sample rate; in +the graphs below, this is simple labeled 0.5 because all frequencies +in the Gaborator are in units of the sample rate. The +lowest-frequency bandpass filter should be centered at, or slightly +below, the lowest frequency of interest to the application at hand. +For example, when analyzing audio, this is often the lower limit of +human hearing; at a sample rate of 44100 Hz, this means 20 Hz / 44100 +Hz ≈ 0.00045. This lower frequency limit is referred to as +the minimum frequency or fmin. +
+ +Although frequencies below fmin are assumed to not be of +interest, they nonetheless need to be preserved to achieve perfect +reconstruction, and that is what the lowpass filter is for. Together, +the lowpass filter and the bandpass filters overlap to cover the full +frequency range from 0 to 0.5.
+ +The spacing of the bandpass filters is specified by the user as an +integer number of filters (or, equivalently, bands) per octave. For +example, when analyzing music, this is often 12 bands per octave (one +band per semitone in the equal-tempered scale), or if a finer +frequency resolution is needed, some multiple of 12.
+ +The following plot shows the frequency responses of the analysis +filters at 12 bands per octave and fmin = 0.03. A more +typical fmin for audio work would be 0.00045, but +that would make the plot hard to read because both the lowpass filter +and the lowest-frequency bandpass filters would be extremely narrow.
+ +
+
+The output of each bandpass filter is shifted down in frequency to +a complex quadrature baseband. The baseband signal is then resampled +at a reduced sample rate, lower than that of the orignal signal but +high enough that there is negligible aliasing given the bandwidth of +the filter in case. The Gaborator uses sample rates related to the +original signal sample rate by powers of two. This means some of +frequency bands are sampled a bit more often than strictly +necessary, but has the advantage that the sampling can be synchronized +to make the samples of many frequency bands coincide in time, which +can be convenient in later analysis or spectrogram rendering. The +complex samples resulting from this process are the spectrogram +coefficients.
+ +The center frequencies of the analysis filters and the points in +time at which they are sampled form a two-dimensional, +multi-resolution time-frequency grid, where high frequencies +are sampled sparsely in frequency but densely in time, and low +frequencies are sampled densely in frequency but sparsely in time.
+ +The following plot illustrates the time-frequency sampling grid +corresponding to the parameters used in the previous plot. Note that +frequency was the X axis in the previous plot, but is the Y axis +here. The plot covers a time range of 128 signal samples, but +conceptually, the grid extends arbitrarily far in time, in both the +positive and the negative direction.
+ +
+
+Resynthesis
+ +Resynthesizing a signal from the coefficients is more or less the +reverse of the analysis process. The coefficients are frequency +shifted from the complex baseband back to their original center +frequencies and run through a reconstruction filter bank +that is a dual of the analysis filter bank. The following +plot shows the frequency responses of the reconstruction filters +corresponding to the analysis filters shown earlier.
+ +
+
+Although the bandpass filters may look similar to the Gaussian +filters of the analysis filter bank, their shapes are actually subtly +different.
+ +Spectrogram Rendering
+ +Rendering a spectrogram image from the coefficients involves +taking the magnitude of each complex coefficient, and then +resampling the resulting multi-resolution grid of magnitudes +into an evenly spaced pixel grid.
+ +Because the coefficient sample rate varies by frequency band, the +resampling required in the horizontal (time) direction also varies. +Typically, the high-frequency bands of an audio spectrogram have more +than one coefficient per pixel and require downsampling (decimation), +some bands in the mid-range frequencies have a one-to-one relationship +between coefficients and pixels, and the low-frequency bands +have more than one pixel per coefficient and require upsampling +(interpolation).
+ + + + + diff --git a/lib/gaborator/doc/realtime.html b/lib/gaborator/doc/realtime.html new file mode 100644 index 0000000..5ea5bec --- /dev/null +++ b/lib/gaborator/doc/realtime.html @@ -0,0 +1,126 @@ + + + + + +Is it real-time?
+ +Several people have asked whether the Gaborator is suitable for +real-time applications. There is no simple yes or no answer to +this question, because there are many different definitions of +"real-time", and the answer will depend the definition. +Below are some answers to the question "is it real-time?" +rephrased in terms of different definitions.
+ +Can it processes a recording in less time than its duration?
+ +Yes. For example, at 48 frequency bands per +octave, a single core of a 2.5 GHz Intel Core i5 CPU can analyze some +10 million samples per second, which is more than 200 times faster +than real time for a single channel of 44.1 kHz audio.
+ +Does it have bounded latency? +Can it start producing output before consuming the entire input? +Will it stream?
+Yes. See the streaming example. + +
Does it have low latency?
+ +Probably not low enough for applications such as live musical +effects. The exact latency depends on factors such as the frequency +range and number of bands per octave, but tends to range between +"high" and "very high". For example, with the parameters used in the +online demo, 48 frequency bands per octave down to 20 Hz, the latency +of the analysis side alone is some 3.5 seconds, and if you do +analysis followed by resynthesis, the total latency will +be almost 13 seconds.
+ +This can be reduced by choosing the analysis parameters for low latency. +For example, if you decrease the number of frequency bands per octave to 12, +and increase the minimum frequency to 200 Hz, the latency +will be about 85 milliseconds for analysis only, and about +300 milliseconds for analysis + resynthesis, but this is +still too much for a live effect.
+ +Any constant-Q spectrum analysis involving low frequencies will +inherently have rather high latency (at least for musically useful +values of Q), because the lowest-frequency analysis filters will have +narrow bandwidths, which lead to long impulse responses. Furthermore, +the Gaborator uses symmetric Gaussian analysis filters that were +chosen for properties such as linear phase and accurate +reconstruction, not for low latency, so the latency will be higher +than what might be achievable with a constant-Q filter bank +specifically designed for low latency.
+ +The latency only affects causal applications, and +arises from the need to wait for the arrival of future input samples +needed to calculate the present output, and not from the time it takes +to perform the calculations. In a non-causal application, +such as applying an effect to a recording, the latency does not apply, +and performance is limited only by the speed of the calculations. +This can lead to the somewhat paradoxical situation that applying an +effect to a live stream causes a latency of several seconds, but +applying the same effect to an entire one-minute recording runs in a +fraction of a second.
+ +In analysis and visualization applications that don't need to +perform resynthesis, it is possible to partly hide the latency by +taking advantage of the fact that the coefficients for the higher +frequencies exhibit lower latency than those for low frequencies. +For example, a live spectrogram display could update the +high-frequency parts of the display before the corresponding +low-frequency parts. Alternatively, low-frequency parts of the +spectrogram may be drawn multiple times, effectively animating +the display of the low-frequency coefficients as they converge to +their final values. This approach can be seen in action in +the Spectrolite +iOS app.
+ +Does it support small blocks sizes?
+ +Yes, but there is a significant performance penalty. +The Gaborator works most efficiently when the signal is processed +in large blocks, preferably 217 samples or more, +corresponding to several seconds of signal at typical audio sample +rates.
+ +A real-time application aiming for low latency will want to +use smaller blocks, for examples 25 to 210 +samples, and processing these will be significantly slower. +For example, as of version 1.4, analyzing a signal in blocks of +210 samples takes roughly five times as much CPU as +analyzing it in blocks of 220 samples.
+ +For sufficiently small blocks, the processing time will exceed the +duration of the signal, at which point the system can no longer be +considered real-time. For example, analyzing a 48 kHz audio +stream on a 2.5 GHz Intel Core i5 CPU, this happens at block sizes +below about 24 = 16 samples.
+ +The resynthesis code is currently less optimized for small block +sizes than the analysis code, so the performance penalty for +resynthesizing small blocks is even greater than for analyzing small +blocks.
+ +Can it process a signal stream of any length?
+ +Not in practice — the length is limited by floating point +precision. At typical audio sample rates, roundoff errors start to +become significant after some hours.
+ +Does it avoid dynamic memory allocation in the audio processing path?
+ +Currently, no — it dynamically allocates both the coefficient data +structures and various temporary buffers.
+ + + + + diff --git a/lib/gaborator/doc/ref/gaborator_h.html b/lib/gaborator/doc/ref/gaborator_h.html new file mode 100644 index 0000000..206901e --- /dev/null +++ b/lib/gaborator/doc/ref/gaborator_h.html @@ -0,0 +1,462 @@ + + + + + +Gaborator reference: gaborator.h
+
+Spectrum Analysis Parameters
+ +A parameters object holds a set of parameters that
+determine the frequency range and resolution of the spectrum
+analysis.
+class parameters {
+
+
+Constructor
++parameters(unsigned int bands_per_octave, + double ff_min, + double ff_ref = 1.0); ++
-
+
bands_per_octave
+ - The number of frequency bands per octave. + Values from 4 to 384 (inclusive) are supported. + +
ff_min
+ - The lower limit of the analysis frequency range, in units of the
+ sample rate. The analysis filter bank will extend low enough in
+ frequency that
ff_minfalls between the two lowest + frequency bandpass filters. + Values from 0.001 to 0.13 are supported.
+ ff_ref
+ - The reference frequency, in units of the sample rate.
+ This allows fine-tuning of the analysis and synthesis filter
+ banks such that the center frequency of one of the filters
+ is aligned with
ff_ref. Ifff_ref+ falls outside the frequency range of the bandpass filter bank, this + works as if the range were extended to include +ff_ref. Must be positive. A typical value + when analyzing music is440.0 / fs, where +fsis the sample rate in Hz. +
+
Comparison
++Comparison operators are provided for compatibility with +standard container classes. The ordering is arbitrary but consistent. +
++bool operator<(const parameters &rhs) const; +bool operator==(const parameters &rhs) const; ++ +
+}; ++ +
Spectrogram Coefficients
+ ++template<class T> class analyzer; ++ +
+A coefs object stores a set of spectrogram coefficients.
+It is a dynamic data structure and will be automatically grown to
+accommodate new time ranges, for example as newly recorded audio is analyzed.
+The template argument T
+must match that of the analyzer (usually float).
+The template argument C is the data type used to store each
+coefficient value; there is usually no need to specify it explicitly as
+it will default to std::complex<T>.
+
+template<class T, class C = std::complex<T>>
+class coefs {
+
+Constructor
++coefs(analyzer<T> &a); ++
+Construct an empty set of coefficients for use with the spectrum
+analyzer a. This represents a signal that is zero
+at all points in time.
+
+}; ++ +
Spectrum Analyzer
+ +
+The analyzer object performs spectrum analysis and/or resynthesis
+according to the given parameters. The template argument T is
+the floating-point type to use for the calculations. This is typically float;
+alternatively, double can be used for increased accuracy at the
+expense of speed and memory consumption.
template<class T>
+class analyzer {
+Constructor
+ ++analyzer(const parameters ¶ms); ++
-
+
params
+ - The spectrum analysis parameters. +
Analysis and synthesis
+ ++void +analyze(const T *signal, + int64_t t0, + int64_t t1, + coefs<T> &coefs) const; ++
Spectrum analyze the samples at *signal and add the
+resulting coefficients to coefs.
+
-
+
signal
+ - The signal samples to analyze, beginning with the sample from time
t0+ and ending with the last sample before timet1, for a total of +t1 - t0samples. + t0
+ - The point in time when the sample at
signal[0]was taken, + in samples. For example, when analyzing an audio recording, this is typically + 0 for the first sample in the recording, but this reference point is arbitrary, + and negative times are valid. Accuracy begins to successively decrease + outside the range of about ±108 samples, so using + large time values should be avoided when they are not necessary because + of the length of the track. +
+ t1
+ - The point in time of the sample one past the
+ end of the array of samples at
signal, + in samples. +
+ coefs- The coefficient object that the results of the + spectrum analysis are added to. +
If the coefs object already contains some
+coefficients, the new coefficients are summed to those already
+present. Because the analysis is a linear operation, this allows a
+signal to be analyzed in blocks, by making multiple calls
+to analyze() with non-overlapping ranges that together
+cover the entire signal. For efficiency, the blocks should
+be large, as in
+analyze(first_131072_samples, 0, 131072, coefs),
+analyze(next_131072_samples, 131072, 262144, coefs),
+etc.
+
+void +synthesize(const coefs<T> &coefs, + uint64_t t0, + uint64_t t1, + T *signal) const; ++
Synthesize signal samples from the coefficients coef and store
+them at *signal.
+
-
+
coefs- The coefficients to synthesize the signal from. +
t0
+ - The point in time of the first sample to synthesize,
+ in samples, using the same time scale as in
analyze().
+ t1
+ - The point in time of the sample one past the last one to synthesize. +
signal
+ - The synthesized signal samples will be written here,
+ beginning with the sample from time
t0and + and ending with the last sample before timet1, + for a total oft1 - t0samples.
+
The time range t0...t1 may extend outside
+the range analyzed using analyze(), in which case the
+signal is assumed to be zero in the un-analyzed range.
A signal may be synthesized in blocks by making multiple calls to
+analyze() with different sample ranges. For efficiency,
+the blocks should be large, and each t0 should
+be multiple of a large power of two.
+ +
Frequency Band Numbering
+ +The frequency bands of the analysis filter bank are numbered by +nonnegative integers that increase towards lower (sic) frequencies. +There is a number of bandpass bands corresponding to the +logarithmically spaced bandpass analysis filters, from near 0.5 +(half the sample rate) to +near fmin, and a single lowpass band containing the +residual signal from frequencies below fmin. +The numbering can be examined using the following methods: +
+ ++int bandpass_bands_begin() const; ++
+Return the smallest valid bandpass band number, corresponding to the +highest-frequency bandpass filter.
++int bandpass_bands_end() const; ++
+Return the bandpass band number one past the highest valid bandpass +band number, corresponding to one past the lowest-frequency bandpass +filter. +
++int band_lowpass() const; ++
+Return the band number of the lowpass band. +
++int band_ref() const; ++
+Return the band number corresponding to the reference frequency
+ff_ref. If ff_ref falls within
+the frequency range of the bandpass filter bank, this will
+be a valid bandpass band number, otherwise it will not.
+
+double band_ff(int band) const; ++
+Return the center frequency of band number band, in units of the +sampling frequency. +
+ +Support
++double analysis_support() const; ++
Returns the one-sided worst-case time domain support of any of the
+analysis filters. When calling analyze() with a sample at time t,
+only spectrogram coefficients within the time range t ± support
+will be significantly changed. Coefficients outside the range may change,
+but the changes will sufficiently small that they may be ignored without
+significantly reducing accuracy.
+double synthesis_support() const; ++
Returns the one-sided worst-case time domain support of any of the
+reconstruction filters. When calling synthesize() to
+synthesize a sample at time t, the sample will only be
+significantly affected by spectrogram coefficients in the time
+range t ± support. Coefficients outside the range may
+be used in the synthesis, but substituting zeroes for the actual
+coefficient values will not significantly reduce accuracy.
+}; ++ +
Functions
+ +Iterating Over Existing Coefficients
+ ++template <class T, class F, class C0, class... CI> +void process(F f, + int b0, + int b1, + int64_t t0, + int64_t t1, + coefs<T, C0> &coefs0, + coefs<T, CI>&... coefsi); ++ +
+Process one or more coefficient sets coefs0... by applying
+the function f to each coefficient present in coefs0,
+in an indeterminate order.
This can be optionally limited to coefficients whose
+band number b and sample time t satisfy
+b0 ≤ b < b1 and
+t0 ≤ t < t1.
+To process every coefficient present
+in coefs0, pass INT_MIN, INT_MAX, INT64_MIN, INT64_MAX
+for the arguments b0, b1, t0,
+and t1, respectively.
+
The function f should have the call signature
+template<class T> +void f(int b, int64_t t, std::complex<T> &c0, std::complex<T> &ci...); ++
where
+-
+
b
+ - The band number of the frequency band the coefficients
+
c0andci...pertain to. + This may be either a bandpass band or the lowpass band.
+ t
+ - The point in time the coefficients
c0and +ci...pertain to, in samples
+ c0
+ - A reference to a complex coefficient from
coefs0
+ ci...
+ - Optional references to complex coefficients from the additional
+ coefficient sets
coefsi....
+
The function f may read and/or modify each of the
+coefficients passed through c0 and each
+ci....
The first coefficient set c0 is a special case when
+it comes to the treatment of missing values. Coefficients missing
+from c0 will not be iterated over at all, but when a
+coefficient is iterated over and is missing from one of the additional
+coefficient sets ci..., it will be automatically created
+and initialized to zero in that additional coefficient set.
Note: The template parameters C0
+and CI... exist to support the processing of coefficient
+sets containing data of types other
+than std::complex<T>, which is not currently part of the
+documented API. In typical use, there is no need to specify them when
+calling apply() because the template parameter list
+can be deduced, but if they are expicitly specified, they should all
+be std::complex<T>.
+
Creating New Coefficients
+ ++template <class T, class F, class C0, class... CI> +void fill(F f, + int b0, + int b1, + int64_t t0, + int64_t t1, + coefs<T, C0> &coefs0, + coefs<T, CI>&... coefsi); ++
+Fill a region of the time-frequency plane with coefficients
+and apply the function f to each.
+
This works like process() except that it is not limited
+to processing coefficients that already exist in coefs0;
+instead, any missing coefficients in coefs0 as well as
+any of the coefsi... are created and initialized to zero
+before f is called.
The t0 and t1 arguments must specify an
+explicit, bounded time range — they must not be given as
+INT64_MIN and/or INT64_MAX as that would mean creating coefficients
+for an an astronomically large time range, requiring a correspondingly
+astronomical amount of memory.
Forgetting Coefficients
++template <class T> +void forget_before(const analyzer<T> &a, + coefs<T> &c, + int64_t limit); ++
Allow the coefficients for points in time before limit
+(a time in units of samples) to be forgotten.
+Streaming applications can use this to free memory used by coefficients
+that are no longer needed. Coefficients that have been forgotten will
+read as zero. This does not guarantee that all coefficients before
+limit are forgotten, only that ones for
+limit or later are not, and that the amount of memory
+consumed by any remaining coefficients before limit is
+bounded.
Legacy API For Iterating Over Existing Coefficients
+ +Prior to version 1.5, the only way to iterate over
+coefficients was the apply() function.
+It is similar to process(), except that it
+
-
+
- requires an additional
analyzerargument, + - takes arguments in a different order, +
- applies a function
ftaking arguments in a different order, + - does not support restricting the processing to a range of band numbers, +
- only supports iterating over a single coefficient set, and +
- provides default values for t0 and t1. +
In new code, process() is preferred.
+template <class T, class F> +void apply(const analyzer<T> &a, + coefs<T> &c, + F f, + int64_t t0 = INT64_MIN, + int64_t t1 = INT64_MAX); ++
+Apply the function f to each coefficient in the coefficient
+set c for points in time t that satisfy
+t0 ≤ t < t1.
+If the t0 and t1 arguments are omitted, f
+is applied to every coefficient.
+
-
+
a
+ - The spectrum analyzer that produced the coefficients
c
+ c
+ - A set of spectrogram coefficients +
f
+ - A function to apply to each coefficient in
c, + with the call signature ++template<class T> +void f(std::complex<T> &coef, int band, int64_t t); +
+-
+
coef
+ - A reference to a single complex coefficient. This may be read and/or modified. +
band
+ - The band number of the frequency band the coefficient
coef0pertains to. + This may be either a bandpass band or the lowpass band.
+ t
+ - The point in time the coefficient
c0pertains to, in samples
+ t0- When not
INT64_MIN, only applyfto the coefficients for time ≥t0
+ t1- When not
INT64_MAX, only applyfto the coefficients for time <t1
+
+
Gaborator reference: API Introduction
+ +The public API of the Gaborator library is defined in the HTML +documentation in the form of annotated C++ declarations. These are +similar to the actual declarations in the respective header files, but +simplified for clarity and omitting implementation details.
+ +The actual implementation in the header file may be different in a
+number of ways but nonetheless compatible with the documented API.
+For example, classes may be declared using the keyword struct
+rather than class, function parameter names may be
+different, types may be declared using different but equivalent
+typedefs, and functions or templates in the header file may have
+additional arguments with default values. Any classes, functions, and
+other definitions not mentioned in the documentation should be
+considered private and are subject to change or deletion without
+notice.
+
All definitions are in the namespace gaborator.
+Applications need to either prefix class names
+with gaborator::, or use using namespace
+gaborator;.
+
Gaborator reference: render.h
+
+Spectrogram Rendering with Power-of-Two Scaling
++template <class OI, class T> +void render_p2scale(const analyzer<T> &a, + const coefs<T> &c, + int64_t xorigin, int64_t yorigin, + int64_t xi0, int64_t xi1, int xe, + int64_t yi0, int64_t yi1, int ye, + OI output); ++
Render a rectangular array of pixel values representing signal +amplitudes in time-frequency space, optionally scaling up or +down by powers of two. +
+-
+
a
+ - The spectrum analyzer that produced the coefficients
c
+ c
+ - A set of spectrogram coefficients to render +
xorigin
+ - The point in time corresponding to pixel X coordinate 0, in samples +
yorigin
+ - The band number of the frequency band corresponding to pixel Y coordinate 0 +
xi0
+ - The X coordinate of the first pixel to render +
xi1
+ - The X coordinate one past the last pixel to render +
xe
+ - The horizontal scaling exponent. One horizontal pixel corresponds to 2xe signal samples. +
yi0
+ - The Y coordinate of the first pixel to render +
yi1
+ - The Y coordinate one past the last pixel to render +
ye
+ - The vertical scaling exponent. One vertical pixel corresponds to 2ye frequency bands. +
output
+ - A random access iterator through which the output
+ pixel amplitude values will be written. This is
+ typically a
float *. A total of +(xi1 - xi0) * (yi1 - yi0))values will be written. +
+
Utility Functions
++template <class T> +unsigned int float2pixel_8bit(T amp); ++
Convert a normalized amplitude value to a 8-bit greyscale pixel value.
+-
+
amp
+ - A floating point value representing a signal amplitude, nominally ranging from 0 to 1 +
Returns an pixel value ranging from 0 to 255 (inclusive), using an +approximation of the sRGB gamma.
+ + + + + diff --git a/lib/gaborator/doc/render.html b/lib/gaborator/doc/render.html new file mode 100644 index 0000000..981f954 --- /dev/null +++ b/lib/gaborator/doc/render.html @@ -0,0 +1,395 @@ + + + + + +Example 1: Rendering a Spectrogram Image
+ +Introduction
+ +This example shows how to generate a greyscale constant-Q +spectrogram image from an audio file using the Gaborator library. +
+ +Preamble
+ +We start off with some boilerplate #includes.
+ ++#include <memory.h> +#include <iostream> +#include <fstream> +#include <sndfile.h> ++ +
The Gaborator is a header-only library — there are no C++ files
+to compile, only header files to include.
+The core spectrum analysis and resynthesis code is in
+gaborator/gaborator.h, and the code for rendering
+images from the spectrogram coefficients is in
+gaborator/render.h.
+#include <gaborator/gaborator.h> +#include <gaborator/render.h> ++ +
The program takes the names of the input audio file and output spectrogram +image file as command line arguments, so we check that they are present:
+ +
+int main(int argc, char **argv) {
+ if (argc < 3) {
+ std::cerr << "usage: render input.wav output.pgm\n";
+ exit(1);
+ }
+
+
+Reading the Audio
+ +The audio file is read using the libsndfile library
+and stored in a std::vector<float>.
+Note that although libsndfile is used in this example,
+the Gaborator library itself does not depend on or
+use libsndfile.
+ SF_INFO sfinfo;
+ memset(&sfinfo, 0, sizeof(sfinfo));
+ SNDFILE *sf_in = sf_open(argv[1], SFM_READ, &sfinfo);
+ if (! sf_in) {
+ std::cerr << "could not open input audio file: "
+ << sf_strerror(sf_in) << "\n";
+ exit(1);
+ }
+ double fs = sfinfo.samplerate;
+ sf_count_t n_frames = sfinfo.frames;
+ sf_count_t n_samples = sfinfo.frames * sfinfo.channels;
+ std::vector<float> audio(n_samples);
+ sf_count_t n_read = sf_readf_float(sf_in, audio.data(), n_frames);
+ if (n_read != n_frames) {
+ std::cerr << "read error\n";
+ exit(1);
+ }
+ sf_close(sf_in);
+
+In case the audio file is a stereo or multi-channel one,
+mix down the channels to mono, into a new std::vector<float>:
+
+ std::vector<float> mono(n_frames);
+ for (size_t i = 0; i < (size_t)n_frames; i++) {
+ float v = 0;
+ for (size_t c = 0; c < (size_t)sfinfo.channels; c++)
+ v += audio[i * sfinfo.channels + c];
+ mono[i] = v;
+ }
+
+
+The Spectrum Analysis Parameters
+ +Next, we need to choose some parameters for the spectrum analysis: +the frequency resolution, the frequency range, and optionally a +reference frequency.
+ +The frequency resolution is specified as a number of frequency +bands per octave. A typical number for analyzing music signals is 48 +bands per octave, or in other words, four bands per semitone +in the 12-note equal tempered scale.
+ +The frequency range is specified by giving a minimum frequency;
+this is the lowest frequency that will be included in the spectrogram
+display.
+For audio signals, a typical minimum frequency is 20 Hz,
+the lower limit of human hearing. In the Gaborator library,
+all frequencies are given in units of the sample rate rather
+than in Hz, so we need to divide the 20 Hz by the sample
+rate of the input audio file: 20.0 / fs.
Unlike the minimum frequency, the maximum frequency is not given +explicitly — instead, the analysis always produces coefficients +for frequencies all the way up to half the sample rate +(a.k.a. the Nyquist frequency). If you don't need the coefficients +for the highest frequencies, you can simply ignore them.
+ +If desired, one of the frequency bands can be exactly aligned with
+a reference frequency. When analyzing music signals, this is
+typically 440 Hz, the standard tuning of the note A4.
+Like the minimum frequency, it is given in
+units of the sample rate, so we pass 440.0 / fs.
The parameters are held in an object of type
+gaborator::parameters:
+
+ gaborator::parameters params(48, 20.0 / fs, 440.0 / fs); ++ +
The Spectrum Analyzer
+ +Next, we create an object of type gaborator::analyzer;
+this is the workhorse that performs the actual spectrum analysis
+(and/or resynthesis, but that's for a later example).
+It is a template class, parametrized by the floating point type to
+use for the calculations; this is typically float.
+Constructing the gaborator::analyzer involves allocating and
+precalculating all the filter coefficients and other auxiliary data needed
+for the analysis and resynthesis, and this takes considerable time and memory,
+so when analyzing multiple pieces of audio with the same
+parameters, creating a single gaborator::analyzer
+and reusing it is preferable to destroying and recreating it.
+ gaborator::analyzer<float> analyzer(params); ++ +
The Spectrogram Coefficients
+ +The result of the spectrum analysis will be a set of spectrogram
+coefficients. To store them, we will use a gaborator::coefs
+object. Like the analyzer, this is a template class parametrized
+by the data type. Because the layout of the coefficients is determined by
+the spectrum analyzer, it must be passed as an argument to the constructor:
+ gaborator::coefs<float> coefs(analyzer); ++ +
Running the Analysis
+ +Now we are ready to do the actual spectrum analysis,
+by calling the analyze method of the spectrum
+analyzer object.
+The first argument to analyze is a float pointer
+pointing to the first element in the array of samples to analyze.
+The second and third arguments are of type
+int64_t and indicate the time range covered by the
+array, in units of samples. Since we are passing the whole file at
+once, the beginning of the range is sample number zero, and the end is
+sample number mono.size(). The fourth argument is a
+reference to the set of coefficients that the results of the spectrum
+analysis will be stored in.
+
+ analyzer.analyze(mono.data(), 0, mono.size(), coefs); ++ +
Rendering an Image
+ +Now there is a set of spectrogram coefficients in coefs.
+To render them as an image, we will use the function
+gaborator::render_p2scale().
+
Rendering involves two different coordinate +spaces: the time-frequency coordinates of the spectrogram +coefficients, and the x-y coordinates of the image. +The two spaces are related by an origin and a scale factor, +each with an x and y component.
+ +The origin specifies the point in time-frequency space that +corresponds to the pixel coordinates (0, 0). Here, we will +use an origin where the x (time) component +is zero (the beginning of the signal), and the y (frequency) component +is the band number of the first (highest frequency) band:
+ ++ int64_t x_origin = 0; + int64_t y_origin = analyzer.bandpass_bands_begin(); ++ +
render_p2scale() supports scaling the spectrogram in
+both the time (horizontal) and frequency (vertical) dimension, but only
+by power-of-two scale factors. These scale factors are specified
+relative to a reference scale of one vertical pixel per frequency band
+and one horizontal pixel per signal sample.
+
+
Although a horizontal scale of one pixel per signal sample is a +mathematically pleasing reference point, this reference scale is not +used in practice because it would result in a spectrogram that is much +too stretched out horizontally. A more typical scale factor might be +210 = 1024, yielding one pixel for every 1024 signal +samples, which is about one pixel per 23 milliseconds of signal at a +sample rate of 44.1 kHz.
++ int x_scale_exp = 10; ++ +
To ensure that the spectrogram will fit on the screen even in the +case of a long audio file, let's auto-scale it down further until +it is no more than 1000 pixels wide:
++ while ((n_frames >> x_scale_exp) > 1000) + x_scale_exp++; ++ +
In the vertical, the reference scale factor of one pixel per +frequency band is reasonable, so we will use it as-is. In other words, +the vertical scale factor will be 20.
++ int y_scale_exp = 0; ++ +
Next, we need to define the rectangular region of the image +coordinate space to render. Since we are rendering the entire +spectrogram rather than a tile, the top left corner of the +rectangle will have an origin of (0, 0). +
+ ++ int64_t x0 = 0; + int64_t y0 = 0; ++ +
The coordinates of the bottom right corner are determined by the
+length of the signal and the number of bands, respectively, taking the
+scale factors into account.
+The length of the signal in samples is n_frames,
+and we get the number of bands as the difference of the end points of
+the range of band numbers:
+analyzer.bandpass_bands_end() - analyzer.bandpass_bands_begin().
+The scale factor is taken into account by right shifting by the
+scale exponent.
+
+ int64_t x1 = n_frames >> x_scale_exp; + int64_t y1 = (analyzer.bandpass_bands_end() - analyzer.bandpass_bands_begin()) >> y_scale_exp; ++ +
The right shift by y_scale_exp above doesn't actually
+do anything because y_scale_exp is zero, but it would be
+needed if, for example, you were to change y_scale_exp to
+1 to get a spectrogram scaled to half the height. You could also make a
+double-height spectrogram by setting y_scale_exp to -1,
+but then you also need to change the
+>> y_scale_exp to
+<< -y_scale_exp since you can't shift by
+a negative number.
+
We are now ready to render the spectrogram, producing
+a vector of floating-point amplitude values, one per pixel.
+Although this is stored as a 1-dimensional vector of floats, its
+contents should be interpreted as a 2-dimensional rectangular array of
+(y1 - y0) rows of (x1 - x0) columns
+each, with the row indices increasing towards lower
+frequencies and column indices increasing towards later
+sampling times.
+
+ std::vector<float> amplitudes((x1 - x0) * (y1 - y0)); + gaborator::render_p2scale( + analyzer, + coefs, + x_origin, y_origin, + x0, x1, x_scale_exp, + y0, y1, y_scale_exp, + amplitudes.data()); ++ +
Writing the Image File
+ +To keep the code simple and to avoid additional library
+dependencies, the image is stored in
+pgm (Portable GreyMap) format, which is simple
+enough to be generated with just a few lines of inline code.
+Each amplitude value in amplitudes is converted into an 8-bit
+gamma corrected pixel value by calling gaborator::float2pixel_8bit().
+To control the brightness of the resulting image, each
+amplitude value is multiplied by a gain; this may have to be adjusted
+depending on the type of signal and the amount of headroom in the
+recording, but a gain of about 15 often works well for typical music
+signals.
+ float gain = 15; + std::ofstream f; + f.open(argv[2], std::ios::out | std::ios::binary); + f << "P5\n" << (x1 - x0) << ' ' << (y1 - y0) << "\n255\n"; + for (size_t i = 0; i < amplitudes.size(); i++) + f.put(gaborator::float2pixel_8bit(amplitudes[i] * gain)); + f.close(); ++ +
Postamble
+
+To make the example code a complete program,
+we just need to finish main():
+
+ return 0; +} ++ +
Compiling
+
+If you are using macOS, Linux, NetBSD, or a similar system, you can build
+the example by running the following command in the examples
+subdirectory.
+You need to have libsndfile is installed and supported by
+pkg-config.
+
+c++ -std=c++11 -I.. -O3 -ffast-math `pkg-config --cflags sndfile` render.cc `pkg-config --libs sndfile` -o render ++ +
Compiling for Speed
+The above build command uses the Gaborator's built-in FFT implementation,
+which is simple and portable but rather slow. Performance can be
+significantly improved by using a faster FFT library. On macOS, you
+can use the FFT from Apple's vDSP library by defining
+GABORATOR_USE_VDSP and linking with the Accelerate
+framework:
+
+c++ -std=c++11 -I.. -O3 -ffast-math -DGABORATOR_USE_VDSP `pkg-config --cflags sndfile` render.cc `pkg-config --libs sndfile` -framework Accelerate -o render ++ +
On Linux and NetBSD, you can use the PFFFT (Pretty Fast FFT) library. +You can get the latest version from +https://bitbucket.org/jpommier/pffft, +or the exact version that was used for testing from gaborator.com: +
+ ++wget http://download.gaborator.com/mirror/pffft/29e4f76ac53b.zip +unzip 29e4f76ac53b.zip +mv jpommier-pffft-29e4f76ac53b pffft ++
Then, compile it:
++cc -c -O3 -ffast-math pffft/pffft.c -o pffft/pffft.o ++
(If you are building for ARM, you will need to add -mfpu=neon to
+both the above compilation command and the ones below.)
PFFFT is single precision only, but it comes with a copy of FFTPACK which can +be used for double-precision FFTs. Let's compile that, too:
++cc -c -O3 -ffast-math -DFFTPACK_DOUBLE_PRECISION pffft/fftpack.c -o pffft/fftpack.o ++
Then build the example and link it with both PFFFT and FFTPACK:
++c++ -std=c++11 -I.. -Ipffft -O3 -ffast-math -DGABORATOR_USE_PFFFT `pkg-config --cflags sndfile` render.cc pffft/pffft.o pffft/fftpack.o `pkg-config --libs sndfile` -o render ++ +
Running
+Running the following shell commands will download a short example
+audio file (of picking each string on an acoustic guitar), generate
+a spectrogram from it as a .pgm image, and then convert
+the .pgm image into a JPEG image:
+
+wget http://download.gaborator.com/audio/guitar.wav +./render guitar.wav guitar.pgm +cjpeg <guitar.pgm >guitar.jpg ++ +
Example Output
+The JPEG file produced by the above will look like this:
+
+
+
+
+
+
diff --git a/lib/gaborator/doc/snr.html b/lib/gaborator/doc/snr.html
new file mode 100644
index 0000000..632c320
--- /dev/null
+++ b/lib/gaborator/doc/snr.html
@@ -0,0 +1,121 @@
+
+
+
+
+
+Example 4: Measuring the Signal-to-Noise Ratio
+ +Introduction
+ +This example measures the signal-to-noise ratio (SNR) of the +resynthesis by analyzing and resynthesizing a test signal +and comparing the resynthesis result to the original. +
+ +Since it does not involve any audio file I/O, this example +does not require the sndfile library, making it the shortest +and simplest one by far.
+ +Preamble
+ ++#include <iostream> +#include <iomanip> +#include <random> +#include <gaborator/gaborator.h> ++ +
Amplitude Measurement
+To calculate the signal-to-noise ratio, we need to measure the
+amplitude of the orignal signal and the error residue. We will use
+the root-mean-square amplitude, which is calculcated by the
+function rms().
+
+double rms(const std::vector<float> &v) {
+ double sqsum = 0;
+ for (size_t i = 0; i < v.size(); i++) {
+ sqsum += v[i] * v[i];
+ }
+ return sqrt(sqsum);
+}
+
+
+Main Program
+ +For the test signal, we use a million samples of white noise with a +uniform amplitude distribution between -1 and +1.
+
+int main(int argc, char **argv) {
+ size_t len = 1000000;
+ std::vector<float> signal_in(len);
+ std::minstd_rand rand;
+ std::uniform_real_distribution<> uniform(-1.0, 1.0);
+ for (size_t i = 0; i < len; i++)
+ signal_in[i] = uniform(rand);
+
+Then we create a spectrum analyzer with 48 bands per octave +and a frequency range of 3 decades (0.0005 to 0.5 times the sample rate):
++ gaborator::parameters params(48, 5e-4); + gaborator::analyzer<float> analyzer(params); ++
...and run the spectrum analyzis:
++ gaborator::coefs<float> coefs(analyzer); + analyzer.analyze(signal_in.data(), 0, len, coefs); ++
...resynthesize the signal into signal_out:
+
+ std::vector<float> signal_out(len); + analyzer.synthesize(coefs, 0, len, signal_out.data()); ++
...measure the resynthesis error:
++ std::vector<float> error(len); + for (size_t i = 0; i < len; i++) + error[i] = signal_out[i] - signal_in[i]; ++
...calculate the signal-to-noise ratio:
++ double snr = rms(signal_in) / rms(error); ++
...and print it in decibels:
++ std::cout << std::fixed << std::setprecision(1) << 20 * log10(snr) << " dB\n"; +} ++
Compiling
+Like Example 1, this example +can be built using a one-line build command: +
++c++ -std=c++11 -I.. -O3 -ffast-math `pkg-config --cflags sndfile` snr.cc `pkg-config --libs sndfile` -o snr ++
Or using the vDSP FFT on macOS:
++c++ -std=c++11 -I.. -O3 -ffast-math -DGABORATOR_USE_VDSP `pkg-config --cflags sndfile` snr.cc `pkg-config --libs sndfile` -framework Accelerate -o snr ++
Or using PFFFT (see Example 1 for how to download and build PFFFT):
++c++ -std=c++11 -I.. -Ipffft -O3 -ffast-math -DGABORATOR_USE_PFFFT `pkg-config --cflags sndfile` snr.cc pffft/pffft.o pffft/fftpack.o `pkg-config --libs sndfile` -o snr ++ +
Running
+The program is run with no arguments:
++./snr ++
This will print the SNR which should be more than 100 dB if the library is working correctly.
+ + + + + diff --git a/lib/gaborator/doc/spectrogram.jpg b/lib/gaborator/doc/spectrogram.jpg new file mode 100644 index 0000000..f854519 Binary files /dev/null and b/lib/gaborator/doc/spectrogram.jpg differ diff --git a/lib/gaborator/doc/stream.html b/lib/gaborator/doc/stream.html new file mode 100644 index 0000000..f270ecc --- /dev/null +++ b/lib/gaborator/doc/stream.html @@ -0,0 +1,279 @@ + + + + + +Example 3: Streaming
+ +Introduction
+ +This example shows how to process streaming audio a block at a time, +rather than operating on a complete recording at once as in the previous +examples.
+ +This program doesn't do anything particulary useful — it just
+inverts the phase of the signal, but not using the obvious method of
+changing the sign of each sample but by changing the sign of each
+spectrogram coefficient. Consider the expression coef =
+-coef a placeholder for your own streaming filter or effect
+code.
Preamble
+ +
+#include <memory.h>
+#include <iostream>
+#include <sndfile.h>
+#include <gaborator/gaborator.h>
+
+int main(int argc, char **argv) {
+ if (argc < 3) {
+ std::cerr << "usage: stream input.wav output.wav\n";
+ exit(1);
+ }
+
+
+Opening the Streams
+ +We again use libsndfile to read the input and write the output. +To keep it simple, this example only handles mono files.
+
+ SF_INFO sfinfo;
+ memset(&sfinfo, 0, sizeof(sfinfo));
+ SNDFILE *sf_in = sf_open(argv[1], SFM_READ, &sfinfo);
+ if (! sf_in) {
+ std::cerr << "could not open input audio file: "
+ << sf_strerror(sf_in) << "\n";
+ exit(1);
+ }
+ if (sfinfo.channels != 1) {
+ std::cerr << "only mono files are supported\n";
+ exit(1);
+ }
+ double fs = sfinfo.samplerate;
+
+ SNDFILE *sf_out = sf_open(argv[2], SFM_WRITE, &sfinfo);
+ if (! sf_out) {
+ std::cerr << "could not open output audio file: "
+ << sf_strerror(sf_out) << "\n";
+ exit(1);
+ }
+
+The next couple of lines work around a design flaw in +libsndfile. By default, when reading a 16-bit +audio file as floating point data and then writing them +as another 16-bit audio file, libsndfile will use slightly +different scale factors on input and output, and the output will +not be bit-identical to the input. To make it easier to verify +that this example actually yields correct results to within +the full 16-bit precision, we select a non-normalized floating +point representation, which does not suffer from this flaw.
++ sf_command(sf_in, SFC_SET_NORM_FLOAT, NULL, SF_FALSE); + sf_command(sf_out, SFC_SET_NORM_FLOAT, NULL, SF_FALSE); ++ +
Spectrum Analysis Parameters
+ +As in Example 1, the parameters are chosen for analyzing music, but +to reduce the latency, the number of frequency bands per octave is reduced +from 48 to 12 (one per semitone), and the lower frequency limit of +the bandpass filter bank is raised from 20 Hz to 200 Hz.
++ gaborator::parameters params(12, 200.0 / fs, 440.0 / fs); + gaborator::analyzer<float> analyzer(params); ++ +
Calculating Latency
+The spectrogram coefficients are calculated by applying symmetric
+FIR filters to the audio signal. This means a spectrogram coefficient
+for any given point in time t is a weighted average of samples
+from both before and after t, representing both past and future
+signal. The width of the filter impulse response depends on the
+bandwidth, which in turn depends on the center frequency of its band.
+The lowest-frequency filters have the narrowest bandwidths, and
+therefore the widest impulses response, and need the greatest amount
+of past and future signal. The width of the filter impulse response
+is called its support, and the worst-case (widest) support of
+any analysis filter can be found by calling the function
+gaborator::analyzer::analysis_support(). This returns
+the one-sided support, the width of the impulse
+response to each side of its center, as a floating point number.
+To be on the safe side, let's round this up to the next integer:
+ size_t analysis_support = ceil(analyzer.analysis_support()); ++
Similarly, when resynthesizing audio from coefficients, calculating
+a sample at time t involves applying symmetric FIR
+reconstruction filters, calculating a weighted average of both past and
+future spectrogram coefficients. The support of the widest reconstruction
+filter can be calculated by calling
+gaborator::analyzer::synthesis_support():
+
+ size_t synthesis_support = ceil(analyzer.synthesis_support()); ++ +
In a real-time application, the need to access future signal +samples and/or coefficients causes latency. A real-time audio +analysis application that needs to examine the coefficients for +time t can only do so when it has received the input samples up +to time t + analysis_support, and therefore has a minimum latency of +analysis_support. A real-time filtering or effect +application, such as the present example, +incurs latency from both analysis and reconstruction +filters, and can only produce the output sample for time t once +it has received the input samples up to +t + analysis_support + synthesis_support, +for a minimum latency of analysis_support + synthesis_support. +Let's print this total latency to standard output: +
++ std::cerr << "latency: " << analysis_support + synthesis_support << " samples\n"; ++ +
In a practical real-time system, there will be additional latency +caused by processing the signal in blocks of samples rather than a +sample at a time. Since the block size is a property of the overall +system, and causes latency even if the Gaborator is not involved, that +latency is considered outside the scope of this discussion. +
+ +Streaming
+To mimic a typical real-time system, the audio is processed
+in fixed-size blocks (here, 1024 samples). If the size
+of the input file is not divisible by the block size, the last block
+is padded with zeroes.
+The variable t_in keeps track of time, indicating
+the sampling time of the first sample of the current input block,
+in units of samples.
+
+ gaborator::coefs<float> coefs(analyzer);
+ const size_t blocksize = 1024;
+ std::vector<float> buf(blocksize);
+ int64_t t_in = 0;
+ for (;;) {
+ sf_count_t n_read = sf_readf_float(sf_in, buf.data(), blocksize);
+ if (n_read == 0)
+ break;
+ if (n_read < blocksize)
+ std::fill(buf.data() + n_read, buf.data() + blocksize, 0);
+
+Now we can spectrum analyze the current block of samples. Note how
+the time range,
+t_in...t_in + blocksize,
+is explicitly passed to analyze().
+
+ analyzer.analyze(buf.data(), t_in, t_in + blocksize, coefs); ++
The call to analyze() updates the coefficients
+for the time range from t_in - analysis_support to
+t_in + blocksize + analysis_support. The oldest
+blocksize samples of this time range,
+that is, from t_in - analysis_support to
+t_in - analysis_support + blocksize, were now updated for
+the last time and will not be affected by future input blocks.
+Therefore, it is now safe to examine and/or modify these
+coefficients as required by your application. Here, by way
+of example, we simply change their signs to invert the phase of the signal.
+Note that unlike the earlier filter example where prorcess()
+applied a function to all the coefficients, here it is applied only to
+the coefficients within a limited time range.
+
+ process(
+ [&](int, int64_t, std::complex<float> &coef) {
+ coef = -coef;
+ },
+ INT_MIN, INT_MAX,
+ t_in - (int)analysis_support,
+ t_in - (int)analysis_support + (int)blocksize,
+ coefs);
+
+
+Next, we will generate a block of output samples. To get correct results,
+we can only generate output when the coefficients that the output samples
+depend on will no longer change. Specifically, a resynthesized audio
+sample for time t will depend on the coefficients of the
+time range t - synthesis_support...t +
+synthesis_support. To ensure that the resynthesis uses only
+coefficients that have already been processed by
+the process() call above, the most recent block of samples
+that can safely be resynthesized ranges from t_out = t_in -
+analysis_support - synthesis_support to t_out +
+blocksize.
+ int64_t t_out = t_in - analysis_support - synthesis_support; + analyzer.synthesize(coefs, t_out, t_out + blocksize, buf.data()); ++
The synthesized audio can now be written to the output file:
+
+ sf_count_t n_written = sf_writef_float(sf_out, buf.data(), blocksize);
+ if (n_written != blocksize) {
+ std::cerr << "write error\n";
+ exit(1);
+ }
+
+Coefficients older than t_out + blocksize - synthesis_support
+will no longer be needed to synthesize the next block of output signal, so
+it's now OK to forget them and free the memory they used:
+
+ forget_before(analyzer, coefs, t_out + blocksize - synthesis_support); ++
This concludes the block-by-block processing loop.
++ t_in += blocksize; + } ++ +
Postamble
++ sf_close(sf_in); + sf_close(sf_out); + return 0; +} ++ +
Compiling
+Like the previous ones, this example can also be built using a one-line build command: +
++c++ -std=c++11 -I.. -O3 -ffast-math `pkg-config --cflags sndfile` stream.cc `pkg-config --libs sndfile` -o stream ++
Or using the vDSP FFT on macOS:
++c++ -std=c++11 -I.. -O3 -ffast-math -DGABORATOR_USE_VDSP `pkg-config --cflags sndfile` stream.cc `pkg-config --libs sndfile` -framework Accelerate -o stream ++
Or using PFFFT (see Example 1 for how to download and build PFFFT):
++c++ -std=c++11 -I.. -Ipffft -O3 -ffast-math -DGABORATOR_USE_PFFFT `pkg-config --cflags sndfile` stream.cc pffft/pffft.o pffft/fftpack.o `pkg-config --libs sndfile` -o stream ++ +
Running
+Running the following shell commands will download an example +audio file containing an impulse (a single sample of maximum amplitude) +padded with silence to a total of 65536 samples, and process it.
++wget http://download.gaborator.com/audio/impulse.wav +./stream impulse.wav impulse_streamed.wav ++ +
The file impulse_streamed.wav will be identical to
+impulse.wav except that the impulse will be of
+opposite polarity, and delayed by the latency of
+analysis_support + synthesis_support samples.
Example 5: Synthesis from Scratch
+ +Introduction
+ +This example demonstrates how to synthesize a signal by creating +spectrogram coefficients from scratch rather than by analyzing an +existing signal. It creates a random pentatonic melody of decaying +sine waves as spectrogram coefficients and then synthesizes audio +from them. +
+ +Preamble
+ +This example program takes a single command line argument, the name +of the output file.
+
+#include <memory.h>
+#include <iostream>
+#include <sndfile.h>
+#include <gaborator/gaborator.h>
+
+int main(int argc, char **argv) {
+ if (argc < 2) {
+ std::cerr << "usage: synth output.wav\n";
+ exit(1);
+ }
+
+
+Synthesis Parameters
+ +Although this example does not perform any analysis, we nonetheless
+need to create an analyzer object, as it is used for both
+analysis and synthesis purposes. To generate the frequencies of the
+12-note equal-tempered scale, we need 12 bands per octave; a multiple
+of 12 would also work, but here we don't need the added frequency
+resolution that would bring, and the time resolution would be
+worse.
To simplify converting MIDI note numbers to band numbers, we choose +the frequency of MIDI note 0 as the reference frequency; this is +8.18 Hz, which happens to be outside the frequency range of the +bandpass filter bank, but that doesn't matter.
+ ++ double fs = 44100; + gaborator::parameters params(12, 20.0 / fs, 8.18 / fs); + gaborator::analyzer<float> analyzer(params); ++ +
Melody Parameters
+ ++We will use the A minor pentatonic scale, which contains the +following notes (using the MIDI note numbering):
+
+ static int pentatonic[] = { 57, 60, 62, 64, 67 };
+
+
++The melody will consist of 64 notes, at a tempo of 120 beats per +minute: +
++ int n_notes = 64; + double tempo = 120.0; + double beat_duration = 60.0 / tempo; ++ +
+The variable volume determines the amplitude of
+each note, and has been chosen such that there will be no clipping
+of the final output.
+
+ float volume = 0.2; ++ +
Composition
+ +We start with an empty coefficient set:
++ gaborator::coefs<float> coefs(analyzer); ++ +
Each note is chosen randomly from the pentatonic scale and added
+to the coefficient set by calling the function fill().
+The fill() function is similar to the process()
+function used in previous examples, except that it can be used to
+create new coefficients rather than just modifying existing ones.
Each note is created by calling fill() on a region of
+the time-frequency plane that covers a single band in the frequency
+dimension and the duration of the note in the time dimension. Each
+coefficient within this region is set to a complex number whose
+magnitude decays exponentially over time, like the amplitude of a
+plucked string. The phase is arbitrarily set to zero by using an
+imaginary part of zero. Since notes can overlap, the new coefficients
+are added to any existing ones using the += operator
+rather than overwriting them.
Note that band numbers increase towards lower frequencies but MIDI
+note numbers increase towards higher frequencies, hence the minus sign
+in front of midi_note.
+
+ for (int i = 0; i < n_notes; i++) {
+ int midi_note = pentatonic[rand() % 5];
+ double note_start_time = beat_duration * i;
+ double note_end_time = note_start_time + 3.0;
+ int band = analyzer.band_ref() - midi_note;
+ fill([&](int, int64_t t, std::complex<float> &coef) {
+ float amplitude =
+ volume * expf(-2.0f * (float)(t / fs - note_start_time));
+ coef += std::complex<float>(amplitude, 0.0f);
+ },
+ band, band + 1,
+ note_start_time * fs, note_end_time * fs,
+ coefs);
+ }
+
+
+Synthesis
+ +We can now synthesize audio from the coefficients by
+calling synthesize(). Audio will be generated
+starting half a second before the first note to allow for the pre-ringing
+of the synthesis filter, and ending a few seconds after the
+last note to allow for its decay.
+
+ double audio_start_time = -0.5; + double audio_end_time = beat_duration * n_notes + 5.0; + int64_t start_frame = audio_start_time * fs; + int64_t end_frame = audio_end_time * fs; + size_t n_frames = end_frame - start_frame; + std::vector<float> audio(n_frames); + analyzer.synthesize(coefs, start_frame, end_frame, audio.data()); ++ +
Writing the Audio
+ +Since there is no input audio file to inherit a file format from,
+we need to choose a file format for the output file by filling in the
+sfinfo structure:
+ SF_INFO sfinfo; + memset(&sfinfo, 0, sizeof(sfinfo)); + sfinfo.samplerate = fs; + sfinfo.channels = 1; + sfinfo.format = SF_FORMAT_WAV | SF_FORMAT_PCM_16; ++ +
The rest is identical to +Example 2: +
+
+ SNDFILE *sf_out = sf_open(argv[1], SFM_WRITE, &sfinfo);
+ if (! sf_out) {
+ std::cerr << "could not open output audio file: "
+ << sf_strerror(sf_out) << "\n";
+ exit(1);
+ }
+ sf_command(sf_out, SFC_SET_CLIPPING, NULL, SF_TRUE);
+ sf_count_t n_written = sf_writef_float(sf_out, audio.data(), n_frames);
+ if (n_written != n_frames) {
+ std::cerr << "write error\n";
+ exit(1);
+ }
+ sf_close(sf_out);
+ return 0;
+}
+
+
+Compiling
+Like Example 1, this example +can be built using a one-line build command: +
++c++ -std=c++11 -I.. -O3 -ffast-math `pkg-config --cflags sndfile` synth.cc `pkg-config --libs sndfile` -o synth ++
Or using the vDSP FFT on macOS:
++c++ -std=c++11 -I.. -O3 -ffast-math -DGABORATOR_USE_VDSP `pkg-config --cflags sndfile` synth.cc `pkg-config --libs sndfile` -framework Accelerate -o synth ++
Or using PFFFT (see Example 1 for how to download and build PFFFT):
++c++ -std=c++11 -I.. -Ipffft -O3 -ffast-math -DGABORATOR_USE_PFFFT `pkg-config --cflags sndfile` synth.cc pffft/pffft.o pffft/fftpack.o `pkg-config --libs sndfile` -o synth ++ +
Running
+The example program can be run using the command
++./synth melody.wav ++
The resulting audio will be in melody.wav.