6.3.2 BeamForming

6.3.2.1 Outline of the node

BeamForming node performs sound source separation based on the following methods:

• DS : Delay-and-Sum beamforming

• WDS : Weighted Delay-and-Sum beamforming

• NULL : NULL beamforming

• ILSE : Indefinite term and Least Square Estimator based beamforming

• LCMV Linearly Constrained Minimum Variance beamforming

• GJ : Linearly Constrained Minimum Variance beamforming, Griffiths-Jim type

• GICA : Geometrically constrained Independent Component Analysis

• GHDSS : Geometrically constrained Higher-order Decorrelation-based Source Separation

Node inputs are:

• Multi-channel complex spectrum of the input acoustic signal,

• Direction of localized sound sources,

• Direction or localized noise sources.

Note outputs are a set of complex spectrum of each separated sound.

6.3.2.2 Necessary files

6.3.2.3 Usage

When to use

This node is used to perform sound source separation on the sound source direction originated using a microphone array. As a sound source direction, either a value estimated by sound source localization or a constant value may be used.

Typical connection

Figure 6.58 shows connection examples of the BeamForming . The node has three inputs as follows:

1. INPUT_FRAMES takes a multi-channel complex spectrum containing the mixture of sounds produced by for example MultiFFT ,

2. INPUT_SOURCES takes the results of sound source localization produced by for example LocalizeMUSIC or ConstantLocalization ,

3. INPUT_NOISE_SOURCES takes the results of sound source localization for noise sources produced by for example LocalizeMUSIC or ConstantLocalization .

The output is the separated signals.

6.3.2.4 Input-output and property of the node

Input

INPUT_FRAMES

: Matrix<complex<float> >  type. Multi-channel complex spectra. Corresponding to the complex spectrum of input waveform from each microphone, the rows correspond to the channels and the columns correspond to the frequency bins.

INPUT_SOURCES

: Vector<ObjectRef>  type. A Vector array of the Source type object in which sound source localization results are stored. Typically, takes the output of SourceIntervalExtender connected to SourceTracker .

INPUT_NOISE_SOURCES

: Vector<ObjectRef>  type. A Vector array of the Source type object where noise source localization results are stored. The input is optional. It enables to avoid lowering performance of beamforming due to noise.

Output

OUTPUT

: Map<int, ObjectRef>  type. A pair containing the sound source ID and the complex spectrum of the separated sound (Vector<complex<float> >  type). Output as many as the number of sound sources .

Parameter

LENGTH

: int  type. Analysis frame length [samples], which must be equal to the values at a preceding node (e.g. AudioStreamFromMic or the MultiFFT ). The default is 512.

ADVANCE

: int  type. Shift length of a frame [samples], which must be equal to the values at a preceding node (e.g. AudioStreamFromMic or the MultiFFT ). The default is 160.

SAMPLING_RATE

: int  type. Sampling frequency of the input waveform [Hz]. The default is 16000.

LOWER_BOUND_FREQUENCY

: int  type. This parameter is the minimum frequency used when separation processing is performed. Processing is not performed for frequencies below this value and the value of the output spectrum is zero then. The user designates a value in the range from 0 to half of the sampling frequency.

UPPER_BOUND_FREQUENCY

: int  type. This parameter is the maximum frequency used when separation processing is performed. Processing is not performed for frequencies above this value and the value of the output spectrum is zero then. LOWER_BOUND_FREQUENCY $<$ UPPER_BOUND_FREQUENCY must be maintained.

TF_CONJ_FILENAME

: string  type. The file name in which the transfer function database of your microphone array is saved. Refer to Section 5.3.1 for the detail of the file format.

INITW_FILENAME

: string  type. The file name in which the initial value of a separation matrix is described. Initializing with a converged separation matrix through preliminary computation allows for separation with good precision from the beginning. If specified, a matrix in INITW_FILENAME is used as an initial separation matrix. Otherwise, the pre-measured separation matrix obtained from a database in TF_CONJ_FILENAME or the separation matrix estimated from a geometric relation is used as an initial separation matrix.

GICA_SS_METHOD

: string  type. Select a stepsize of ICA (Independent Component Analysis) calculation method based for a blind source separation. If FIX, GICA_SS_MYU designed fix value is used. If LC_MYU, GICA_LC_MYU associated with the step size based on geometric constraint is used. If ADAPTIVE, the stepsize is adaptively determined.

GICA_SS_MYU

: float  type. Designate the stepsize to be used when updating a separation matrix based on blind source separation. The default value is 0.001. If GICA_SS_METHOD = FIX, GICA_SS_MYU is the designated value for the stepsize. If GICA_SS_METHOD = LC_MYU, this parameter is ignored. If GICA_SS_METHOD = ADAPTIVE, the adaptive stepsize is multiplied by GICA_SS_MYU, resulting the final stepsize. By setting this value and GICA_LC_MYU to zero and passing a separation matrix of delay-and-sum beamformer type as INITW_FILENAME, processing equivalent to delay-and-sum beamforming is performed.

GHDSS_SS_METHOD

: string  type. Select a stepsize calculation method based for HDSS (Higher-order Decorrelation-based Source Separation). If FIX, GHDSS_SS_MYU designed fix value is used. If LC_MYU, GHDSS_LC_MYU associated with the step size based on geometric constraint is used. If ADAPTIVE, the stepsize is adaptively determined.

GHDSS_SS_MYU

: float  type. Designate the stepsize to be used when updating a separation matrix based on HDSS (Higher-order Decorrelation-based Source Separation). The default value is 0.001. If GHDSS_SS_METHOD = FIX, GHDSS_SS_MYU is the designated value for the stepsize. If GHDSS_SS_METHOD = LC_MYU, this parameter is ignored. If GHDSS_SS_METHOD = ADAPTIVE, the adaptive stepsize is multiplied by GHDSS_SS_MYU, resulting the final stepsize. By setting this value and GHDSS_LC_MYU to zero and passing a separation matrix of delay-and-sum beamformer type as INITW_FILENAME, processing equivalent to delay-and-sum beamforming is performed.

LCMV_LC_METHOD

: string  type. Select a stepsize calculation method for separation based on linearly constraints. This parameter affects the stepsize for the source separation based on linearly constraints (LC). If FIX, LCMV_LC_MYU designed fix value is used. If ADAPTIVE, the stepsize is adaptively determined.

LCMV_LC_MYU

: float  type. Designate the stepsize to be used when updating a separation matrix based on linearly constraints. The default value is 0.001. If LCMV_LC_METHOD = FIX, LCMV_LC_MYU is the designated value for the stepsize. If LCMV_LC_METHOD = ADAPTIVE, the adaptive stepsize is multiplied by LCMV_LC_MYU, resulting the final stepsize.

GJ_LC_METHOD

: string  type. Select a stepsize calculation method for separation based on linearly constraints. If FIX, GJ_LC_MYU designed fix value is used. If ADAPTIVE, the stepsize is adaptively determined.

GJ_LC_MYU

: float  type. Designate the stepsize to be used when updating a separation matrix based on linearly constraints. The default value is 0.001. If GJ_LC_METHOD = FIX, GJ_LC_MYU is the designated value for the stepsize. If GJ_LC_METHOD = ADAPTIVE, the adaptive stepsize is multiplied by GJ_LC_MYU, resulting the final stepsize.

GICA_LC_METHOD

: string  type. Select a stepsize calculation method for separation based on geometric constraints (GC). If FIX, GICA_LC_MYU designed fix value is used. If ADAPTIVE, the stepsize is adaptively determined.

GICA_LC_MYU

: float  type. Designate the stepsize to be used when updating a separation matrix based on geometric constraints. The default value is 0.001. If GICA_LC_METHOD = FIX, GICA_LC_MYU is the designated value for the stepsize. If GICA_LC_METHOD = ADAPTIVE, the adaptive stepsize is multiplied by LCMV_LC_MYU, resulting the final stepsize. By setting this value and GICA_SS_MYU to zero and passing a separation matrix of delay-and-sum beamformer type as INITW_FILENAME, processing equivalent to delay-and-sum beamforming is performed.

GHDSS_LC_METHOD

: string  type. Select a stepsize calculation method for separation based on geometric constraints. This parameter affects the stepsize for the source separation based on geometric constraints (GC). If FIX, GHDSS_LC_MYU designed fix value is used. If ADAPTIVE, the stepsize is adaptively determined.

GHDSS_LC_MYU

: float  type. Designate the stepsize to be used when updating a separation matrix based on geometric constraints. The default value is 0.001. If GHDSS_LC_METHOD = FIX, GHDSS_LC_MYU is the designated value for the stepsize. If GHDSS_LC_METHOD = ADAPTIVE, the adaptive stepsize is multiplied by GHDSS_LC_MYU, resulting the final stepsize. By setting this value and GHDSS_SS_MYU to zero and passing a separation matrix of delay-and-sum beamformer type as INITW_FILENAME, processing equivalent to delay-and-sum beamforming is performed.

GHDSS_LC_CONST

: string  type. Select geometric constraint method. If DIAG, use only diagonal components (direct sound components) for geometric constraints. If FULL, use off-diagonal components in addition to diagonal components (direct sound components) for geometric constraints. Highly precise separation is possible even with DIAG, because the blind spot is automatically formed by higher order decorrelation.

GICA_SS_SCAL

: float  type. Designate the scale factor of a hyperbolic tangent function (tanh) in calculation of the higher-order correlation matrix. The default value is 1.0. A positive real number greater than zero must be designated. The smaller the value is, the less non-linearity, which makes the calculation close to a normal correlation matrix calculation.

GHDSS_SS_SCAL

: float  type. Designate the scale factor of a hyperbolic tangent function (tanh) in calculation of the higher-order correlation matrix. The default value is 1.0. A positive real number greater than zero must be designated. The smaller the value is, the less non-linearity, which makes the calculation close to a normal correlation matrix calculation.

GHDSS_NOISE_FLOOR

: float  type. Specify the amplitude threshold (upper limit) that regards the input signal as noise. The default value is 0. When the amplitude of the input signal is less than this value, it is regarded as a noise section and the separation matrix is not updated. Specify a positive real number, if the noise is large and the separation matrix does not converge stably.

GHDSS_UPDATE

: string  type. Decide how to update separation matrix. If STEP, update based on geometric constraints is performed after updating based on higher order decorrelation. If TOTAL, perform updates based on higher order decorrelation and update based on geometric constraints at the same time.

UPDATE_METHOD_W

: string  type. Recalculation of separation matrix is required when sound source position information changes. At this time, designate a method to consider sound source position information changed. The separating matrix is internally stored together with the coordinates of the sound source ID and the sound source direction for a certain period of time, and once the sound is stopped, if a sound judged to be a sound source from the same direction is detected, the separating matrix saved again The separation process is performed. At this time, a criterion as to whether or not to update the separation matrix is set. If ID, it is judged by the sound source ID whether it is the same direction sound source or not. If POS, it is judged by comparing the coordinates of the sound source direction. If ID_POS, at first sound source IDs are compared, if they are not determined to be identical, further, judgment is made by comparing the coordinates of the sound source direction.

UPDATE_ACCEPT_DISTANCE

: float  type. Distance to be regarded as the same sound source for movement of the sound source [mm]. If it is within the set distance range, calculation is performed using the updated separation matrix. The default value is 300.0.

EXPORT_W

: bool  type. The user determines if the results of the separation matrix updated will be output. When true, select EXPORT_W_FILENAME.

EXPORT_W_FILENAME

: string  type. Designate the name of the file into which a separation matrix will be output. For its format, see Section 5.3.2.

BF_METHOD

: string  type. Designate the sound source separation method. Currently, this node supports the following separation methods:

• DS : Delay-and-Sum beamforming [1]

• WDS : Weighted Delay-and-Sum beamforming [1]

• NULL : NULL beamforming [1]

• ILSE : Iterative Least Squares with Enumeration [2]

• LCMV : Linearly Constrained Minimum Variance beamforming [3]

• GJ : Griffiths-Jim beamforming [4]

• GICA : Geometrically constrained Independent Component Analysis [5]

• GHDSS : Geometrically constrained Higher-order Decorrelation-based Source Separation [5]

ENABLE_DEBUG

: bool  type. Setting the value to true outputs the separation status to the standard output. The default value is false.

6.3.2.5 Details of the node

Technical details: Basically, the technical detail of each separation method can be found in the references below.

Bried explanation of sound source separation:

Table 6.48 shows the notation of variables used in sound source separation problems. Since the source separation is performed frame-by-frame in the frequency domain, all the variable is computed in a complex field. Also, the separation is performed for all $K$ frequency bins ($1 \leq k \leq K$). Here, we omit $k$ from the notation. Let $N$, $M$, and $f$ denote the number of sound sources and the number of microphones, and the frame index, respectively.

We use the following linear model for the signal processing:

$\displaystyle \boldsymbol {X}(f) $

$\displaystyle = $

$\displaystyle \boldsymbol {H}\boldsymbol {S}(f) + \boldsymbol {N}(f)~ .\label{eq:beamforming-observation} $

(43)

The purpose of the separation is to estimate $\boldsymbol {W}(f)$ based on the following equation:

$\displaystyle \boldsymbol {Y}(f) $

$\displaystyle = $

$\displaystyle \boldsymbol {W}(f)\boldsymbol {X}(f) \label{eq:Beamforming-separation} $

(44)

so that $\boldsymbol {Y}(f)$ is getting closer to $\boldsymbol {S}(f)$. After separation, the estimated $\boldsymbol {W}(f)$ can be saved by setting EXPORT_W=true and put a certain name in EXPORT_W_FILENAME.

TF_CONJ_FILENAME specifies the transfer function matrix $\boldsymbol {H}$ which is pre-measured or pre-calculated. Hereinafter, we denote this pre-measured transfer function as $\hat{\boldsymbol {H}}$ to distinguish from $\boldsymbol {H}$.

Separation by BF_METHOD = DS,WDS,NULL,ILSE: $\boldsymbol {W}(f)$ is directly determined using $\hat{\boldsymbol {H}}$ and corresponding the directions of target recieved at the INPUT_SOURCES terminal and the directions of noise recieved at the INPUT_NOISE_SOURCES terminal.

Separation by BF_METHOD = LCMV,GJ: The cont function $J_{\textrm{L}}(\boldsymbol {W}(f))$ for updating the separation matrix is defined by the directions of target recieved at the INPUT_SOURCES terminal and the directions of noise recieved at the INPUT_NOISE_SOURCES terminal. The equation for updating the separation matrix is described simply as follows:

$\displaystyle \boldsymbol {W}(f+1) $

$\displaystyle = $

$\displaystyle \boldsymbol {W}(f) + \mu \nabla _{\boldsymbol {W}}\boldsymbol {J}_{\textrm{L}}(\boldsymbol {W})(f)~ ,\label{eq:LCMV-GJ-J} $

(45)

where $\nabla _{\boldsymbol {W}}\boldsymbol {J}_{\textrm{L}}(\boldsymbol {W}) = \frac{\partial \boldsymbol {J}_{\textrm{L}}(\boldsymbol {W})}{\partial \boldsymbol {W}}$. LC_MYU specifies the value of $\mu $. If LC_METHOD = ADAPTIVE, this node computes the adaptive stepsize based on the following equation.

$\displaystyle \mu $

$\displaystyle = $

$\displaystyle \left. \frac{\boldsymbol {J}_{\textrm{L}}(\boldsymbol {W})}{\left| \nabla _{\boldsymbol {W}}\boldsymbol {J}_{\textrm{L}}(\boldsymbol {W})\right|^2} \right|_{\boldsymbol {W} = \boldsymbol {W}(f)}\label{eq:LCMV-GJ-mu} $

(46)

Separation by BF_METHOD = GICA: The cont function $J_{\textrm{G}}(\boldsymbol {W}(f))$ for updating the separation matrix is defined by the directions of target recieved at the INPUT_SOURCES terminal and the directions of noise recieved at the INPUT_NOISE_SOURCES terminal.

$\displaystyle J_{\textrm{G}}(\boldsymbol {W}(f)) $

$\displaystyle = $

$\displaystyle J_{\textrm{SS}}(\boldsymbol {W}(f)) + J_{\textrm{LC}}(\boldsymbol {W}(f))~ , \label{eq:GHDSS-J} $

(47)

where $J_{\textrm{SS}}(\boldsymbol {W}(f))$ is the cost function for the blind source separation, $J_{\textrm{LC}}(\boldsymbol {W}(f))$ is the cost function for the source separation based on geometric constraints. The equation for updating the separation matrix is described simply as follows:

$\displaystyle \boldsymbol {W}(f+1) $

$\displaystyle = $

$\displaystyle \boldsymbol {W}(f) + \mu _{\textrm{SS}} \nabla _{\boldsymbol {W}}\boldsymbol {J}_{\textrm{SS}}(\boldsymbol {W})(f) + \mu _{\textrm{LC}} \nabla _{\boldsymbol {W}}\boldsymbol {J}_{\textrm{LC}}(\boldsymbol {W})(f)~ , \label{eq:GHDSS-W} $

(48)

where $\nabla _{\boldsymbol {W}}$ means the partial derivative in respect of $\boldsymbol {W}$ same as Eq. (45). The $\mu _{\textrm{SS}}$ and $\mu _{\textrm{LC}}$ in the equation can be specified by SS_MYU and LC_MYU, respectively. If SS_METHOD = ADAPTIVE, $\mu _{\textrm{SS}}$ is adaptively determined by

$\displaystyle \mu _{\textrm{SS}} $

$\displaystyle = $

$\displaystyle \left. \frac{\boldsymbol {J}_{\textrm{SS}}(\boldsymbol {W})}{\left| \nabla _{\boldsymbol {W}}\boldsymbol {J}_{\textrm{SS}}(\boldsymbol {W})\right|^2} \right|_{\boldsymbol {W} = \boldsymbol {W}(f)}~ .\label{eq:GHDSS-SS-mu} $

(49)

If LC_METHOD = ADAPTIVE, $\mu _{\textrm{LC}}$ is adaptively determined by

$\displaystyle \mu _{\textrm{LC}} $

$\displaystyle = $

$\displaystyle \left. \frac{\boldsymbol {J}_{\textrm{LC}}(\boldsymbol {W})}{\left| \nabla _{\boldsymbol {W}}\boldsymbol {J}_{\textrm{LC}}(\boldsymbol {W})\right|^2} \right|_{\boldsymbol {W} = \boldsymbol {W}(f)}~ .\label{eq:GHDSS-LC-mu} $

(50)

Trouble shooting: Basically, follow the trouble shooting of the GHDSS node.

6.3.2.6 References

• H. Krim and M. Viberg, ’Two decades of array signal processing research: the parametric approach’, in IEEE Signal Processing Magazine, vol. 13, no. 4, pp. 67–94, 1996. D. H. Johnson and D. E. Dudgeon, Array Signal Processing: Concepts and Techniques, Prentice-Hall, 1993.

• S. Talwar, et al.: ’Blind separation of synchronous co-channel digital signals using an antenna array. I. Algorithms’, IEEE Transactions on Signal Processing, vol. 44 , no. 5, pp. 1184 - 1197.

• O. L. FrostIII, ’An Algorithm for Lineary Constrained Adaptive array processing’, Proc. of the IEEE, Vol. 60, No.8, 1972

• L. Griffiths and C. Jim, ’An alternative approach to linearly constrained adaptive beamforming’, IEEE trans. on ant. and propag. Vol. AP-30, No.1, 1982

• H. Nakajima, et al.: ’Blind Source Separation With Parameter-Free Adaptive Step-Size Method for Robot Audition’, IEEE Trans. ASL Vol.18, No.6, pp.1476-1485, 2010.