Bayesian Source Separation

x(t) = A s(t)

Background (top)

The problem of source separation is a ubiquitous problem in the sciences where there are multiple signal sources s(t), which are recorded by multiple detectors. In this scenario, each detector records a mixture x(t) of the original signals. The goal is to recover estimates of the original signals.

The problem of source separation is by its very nature an inductive inference problem. There is not enough information to deduce the solution, so one must use any available information to infer the most probable solution. This information comes in two forms: the signal model and the probability assignments. By adopting a signal model appropriate for the problem, one can develop a specially-tailored algorithm. Many people like the idea of a general blind source separation algorithm that can be applied anywhere. However, since the quality of the results depends on the information put into the algorithm, one will do better with an algorithm that incorporates more specific knowledge.

What I appreciate about the Bayesian approach is that it requires one to make the assumptions explicit. This is not case the case with ad hoc source separation algorithms, which are almost impossible to modify intelligently if they do not quite work for a particular application. With a Bayesian solution, one needs only to trace the problem back to the model, the probability assignments, or a simplifying assumption and modify it appropriately. While this is often easier said than done, it is still better than the situation that one is in when dealing with an ad hoc algorithm where the model and assumptions are implicit and often unknown.

Bayesian ICA (top)

Since our first papers introducing Bayesian source separation in 1997, 1998 and 1999, we have been involved in developing new techniques for separating mixed signals and applying them to a variety of problems.

Our early works considered the Infomax Independent Component Analysis (ICA) algorithm developed by Bell and Sejnowski, which was a neural network based source separation algorithm. We worked to re-cast the problem as an inference problem where the machinery of Bayesian inference could be employed to accommodate additional prior information.

Knuth K.H. 1998. Difficulties applying recent blind source separation techniques to EEG and MEG. In: G.J. Erickson, J.T. Rychert and C.R. Smith (eds.), Maximum Entropy and Bayesian Methods, Boise 1997, Kluwer, Dordrecht, pp. 209-222. arXiv:1501.05068 []
Knuth K.H. 1998. Bayesian source separation and localization. In: A. Mohammad-Djafari (ed.), SPIE'98 Proceedings: Bayesian Inference for Inverse Problems, San Diego, July 1998, pp. 147-158. arXiv:physics/0205069 []
Knuth K.H. 1999. A Bayesian approach to source separation. In: J.-F. Cardoso, C. Jutten and P. Loubaton (eds.), Proceedings of the First International Workshop on Independent Component Analysis and Signal Separation: ICA'99, Aussois, France, Jan. 1999, pp. 283-288. arXiv:physics/0205032 []

Source Separation and Localization (top)

In this picture, the sensors around the bridge recorded sounds from each of the characters during one of the crew's weekly catastrophic events. Since we know that the Starship Enterprise officers won't wander far from their posts, we can use their approximate locations to help separate their recorded speech signals from the mayhem.

In the SPIE98 paper below, I considered an example where the source positions are known with some accuracy. This combined with the propagation laws of the signal (inverse square) leads to a prior probability on the values of the mixing matrix, which, in general, improves the separation. The results aren't perfect however, because I use a prior on the source amplitude histograms that is inappropriate for some of the other recorded signals, such as the photon torpedo blast. These difficulties are discussed in the MaxEnt97 paper above, although in a different context. More detailed information can be found at my old BSE site, and in the papers below. I won't tell you who survived, but its a sure bet that Ensign Jones is toast.

Knuth K.H. 1998. Bayesian source separation and localization. In: A. Mohammad-Djafari (ed.), SPIE'98 Proceedings: Bayesian Inference for Inverse Problems, San Diego, July 1998, pp. 147-158. arXiv:physics/0205069 []

Neural Source Estimation (top)

The Neural Source Estimation problem
(And proof that I have a brain)

Neural activity in the brain results in the generation of both electric currents and magnetic fields. Electric currents flowing through the volume of the brain can be detected using electrodes on the scalp (or capacitors above the scalp) in a technique called electroencephalography (EEG). On the other hand, magnetic fields can be detected using superconducting quantum interference devices (SQUIDs) in a technique called magnetoencephalography (MEG).

For any given stimulus, multiple areas in the brain respond. This results in multiple neural sources each generating electric currents and magnetic fields, a linear superposition of which is recorded by the detectors (EEG and/or MEG). This linear mixing of source signals results in a classic source separation problem.

In our MaxEnt 1998 paper, we explored the possibility of simultaneously performing source separation and source localization by modeling the neural sources as current dipoles.

Knuth K.H., Vaughan H.G., Jr. 1999. Convergent Bayesian formulations of blind source separation and electromagnetic source estimation. In: W. von der Linden, V. Dose, R. Fischer and R. Preuss (eds.), Maximum Entropy and Bayesian Methods, Munich 1998, Dordrecht. Kluwer, pp. 217-226. arXiv:1501.05069 []

In later works, we considered the fact that neural sources exhibit some variability in the timing of their responses to stimuli. We realized that in some cases we could use the fact that different neural sources vary differently in latency (differential variability) to aid in separating the neural responses from different sources. This gave rise to a series of papers that resulted in an algorithm called differentially Variable Component Analysis (dVCA).

The dVCA algorithm is is a highly specialized algorithm that takes into account the fact that EEG/MEG experiments record data in a finite number of experimental trials. Since the activity produced by neural ensembles varies from trial to trial, our signal model accounts for this by allowing the source waveshape to vary in both amplitude and latency. These effects are estimated for each trial along with the stereotypic source waveshape. The relevant publications are below:

Knuth K.H., Shah A.S., Truccolo W., Ding M., Bressler S.L., Schroeder C.E. 2006. Differentially Variable Component Analysis (dVCA): Identifying Multiple Evoked Components using Trial-to-Trial Variability. J Neurophysiol. 95: 3257-3276. doi:10.1152/jn.00663.2005 [PubMed Link] [pdf (799K)]
Truccolo W.A., Knuth K.H., Shah A.S., Bressler S.L., Schroeder C.E., and Ding M. 2003. Estimation of single-trial multi-component ERPs: Differentially variable component analysis. Biol. Cybern. 89(6): 426-38. [PubMed Link] [pdf (717kb)]
Shah A.S., Knuth K.H., Lakatos P., Schroeder C.E. 2003. Lessons from applying differentially variable component analysis (dVCA) to electroencephalographic activity. In: G.J. Erickson, Y. Zhai (eds.), Bayesian Inference and Maximum Entropy Methods in Science and Engineering, Jackson Hole WY 2003, AIP Conference Proceedings 707, American Institute of Physics, Melville NY, pp. 167-181. [pdf (445 kb)]
Shah A.S., Knuth K.H., Truccolo W.A., Ding M., Bressler S.L., Schroeder C.E. 2002. A Bayesian approach to estimating coupling between neural components: evaluation of the multiple component event related potential (mcERP) algorithm. In: C. Williams (ed.), Bayesian Inference and Maximum Entropy Methods in Science and Engineering, Moscow ID 2002, AIP Conference Proceedings 659, American Institute of Physics, Melville NY, pp. 23-38. [pdf (1.05 mb)]
Truccolo, W.A, Ding M., Knuth K.H., Nakamura, R. and Bressler S.L. 2002. Variability of cortical evoked responses: implications for the analysis of functional connectivity. Clinical Neurophysiol. 113(2):206-26. [PubMed link] [pdf (433 kb)]
Truccolo, W.A., Knuth K.H., Ding M., Bressler S.L. 2001. Bayesian estimation of amplitude, latency and waveform of single trial cortical evoked components. In: R.L. Fry and M. Bierbaum (eds.), Bayesian Inference and Maximum Entropy Methods in Science and Engineering, Baltimore 2001, AIP Conference Proceedings 617, American Institute of Physics, Melville NY, pp. 64-73. [pdf (100 kb)]
Knuth K.H., Truccolo, W.A., Bressler S.L., Ding M. 2001. Separation of multiple evoked responses using differential amplitude and latency variability. Proceedings of the Third International Workshop on Independent Component Analysis and Blind Signal Separation (ICA 2001), San Diego CA. arXiv:physics/0204085 [] [pdf (248 kb)]

Informed Source Separation (top)

The Bayesian approach to the source separation problem requires the designer to explicitly describe the signal model in addition to any other information or assumptions that go into the problem description. This leads naturally to the concept of informed source separation, where the algorithm design incorporates relevant information about the specific problem. This approach, which is at the opposite end of the spectrum to blind source separation, promises to enable researchers to design their own high-quality algorithms that are specifically tailored to the problem at hand.

Knuth K.H. 2005. Informed source separation: A Bayesian tutorial. (Invited paper) B. Sankur , E. Çetin, M. Tekalp , E. Kuruoğlu (ed.), Proceedings of the 13th European Signal Processing Conference (EUSIPCO 2005), Antalya, Turkey. arXiv:1311.3001 [stat.ML]