Bayesian Data Analysis
Fall 2018

PHY/CSI/INF 451/551
Class Nos:5616, 5617, 5621, 5622, 6102, 6207
Lecture: TTh 1:15PM - 2:35PM, PH 225

Prof. Kevin H. Knuth
Physics Department
University at Albany
Albany NY USA


There is an Extra Credit HW Assignment available for ALL students in the class. It consists of three problems. Each problem will be worth extra credit (despite the instructions in the old pdf, which I do not feel like rewriting at the moment). You may need to go through some of this material on multidimensional problems and covariance matrices. Each problem will be graded on a scale of 1-100pts. Those points will be added to your total HW points before computing the average HW score.

Matlab code from the Freely Falling Object lecture (Oct 18) has been posted online.

UAlbany MATLAB License: WikiPage with Instructions HERE

Textbook: Data Analysis: A Bayesian Tutorial by Sivia and Skilling, 2nd Edition

Additional Texts (Recommended):
Bayesian Probability Theory: Applications in the Physical Sciences

Last Day to Drop ('W' Assigned):
Graduate Students: Oct 23.
Undergraduate Students: Nov 6

Code for a Nested Sampling variant called MultiNest can be found here. It has some nice examples and is relatively easy to use. Note that if you use MutliNest? (for your projects or otherwise) the authors request that you reference the following papers:
Note that you will have to obtain the full proper citation information.

The powerpoint slides I showed in class were from the short course: Applied Bayesian Inference. Find these in the course Resources.



Note: Readings with links in (parentheses) are optional.

Sep 6
  AI researchers allege that machine learning is alchemy, Science, 2018 (link)
  Frequentist vs Bayesian, Analytics India Magazine, 2018 (link)
  Mike Lee Williams on Probabilistic Programming, ... PyMC3, InfoQ, 2018 (link)

Sep 11
  Cox, Probability, Frequency, and Reasonable Expectation C46
  Knuth & Skilling, Foundations of Inference KS12
  Skilling & Knuth, Measure, Probability, Quantum (SK18)
Sep 6
  Wigner, The Unreasonable Effectiveness of Mathematics in the Natural Sciences, (W60)
  Knuth, The Deeper Roles of Mathematics in Physical Laws, K15
Aug 28
  Ch. 1, Sivia and Skilling
  Jaynes, Bayesian Methods: General Background, J80



Instructor:Dr. Kevin H. Knuth, Associate Professor of Physics and Informatics PH 211, 442-4501
Office Hours:Thursday 2:45pm – 3:45pm, PH 211
TAs:Bertrand Carado, Friday 10:00am - 11:00am, PH 228
 Bertrand Carado, Friday 11:00am - 12:00pm, PH 224
 Siqian Zhao, Wednesday 10:00am - 11:00am, UAB434
Required Text:Data Analysis: A Bayesian Tutorial by Sivia and Skilling, 2nd Edition
Required License:MatLab Release 14 or Later: Student Edition
Additional Texts (Recommended):Bayesian Probability Theory: Applications in the Physical Sciences

Course Description: Introduction to both the principles and practice of Bayesian and maximum entropy methods for data analysis, signal processing, and machine learning. This is a hands-on course that will introduce the use of the MATLAB computing language for software development. Students will learn to write their own Bayesian computer programs to solve problems relevant to physics, chemistry, biology, earth science, and signal processing, as well as hypothesis testing and error analysis. Optimization techniques to be covered include gradient ascent, fixed-point methods, and Markov chain Monte Carlo sampling techniques.
3 credits.
Prerequisite(s): A MAT 214 (or equivalent) and A CSI 101 or A CSI 201.

Course Objectives: Learn how to use the sum and product rules of probability to compute probabilities of various hypotheses. Learn how to use Bayes theorem to solve inference-based data analysis problems. Learn how to assign prior probabilities and likelihood functions based on the problem at hand. Learn both analytic and numerical techniques for computing the mean and mode of a probability density function as well as the accompanying uncertainties and the Bayesian evidence.

1Aug 28IntroductionCh. 1, J80
 Aug 30Problem Solving 
2Sep 4MATLAB Overview HW1p:Sep 13
 Sep 6Foundations of Inference(W60),K15
3Sep 11Foundations of InferenceC46,KS12, (GKS10)
 Sep 13Bayes Theorem ExamplesCh. 2
4Sep 18Bayes Theorem Examples HW2w:Sep 27
 Sep 20Bayes Theorem ExamplesCh. 3
5Sep 25Probability Density Funcs 
 Sep 27Moments of Distributions HW3p:Oct 16
6Oct 2Quantifying UncertaintyChs. 4 & 5HW4w:Oct 9
 Oct 4Assigning Probs and EntropyJ68 Gif07
7Oct 9Marginalization 
 Oct 11Ex: Length of a Pen 
8Oct 16Ex: Falling Object 
 Oct 18Ex: Histogram BinningK05a 
9Oct 23Review 
 Oct 25Estimation in Many Dims 
10Oct 30MIDTERM 
 Nov 1Ex: Source SeparationK99 
11Nov 6Optimization TechniquesCh 9,10
 Nov 8Sampling from PDFs HW5p:Nov 26
12Nov 13Model Testing and EvidenceKHMMP15 
 Nov 15Nested Sampling 
13Nov 20Nested Sampling 
 Nov 22ThanksgivingPROPOSALS NOV 27
14Nov 27Example: Exoplanet CharacterizationPKA14HW6:Dec 12
 Nov 29Example: Model Testing
15Dec 4Metropolis Hastings MCMCCh. 7
 Dec 6Experimental DesignL03, KC10 
12pm-2pm in PH 225

This course has a website that you can check for updates to the schedule or for special announcements.

Written Homework
Written homework assignments will be assigned approximately weekly. Typically, the solutions will be due 7 days after being assigned. All written homework assignments are expected to be completed and written in a neat and professional manner.

Each Written Homework assignment can be turned in by the end of the day 11:59 pm of the Submission Date above for 100 points. An assignment can be turned in from 1-3 days after the Submission Date for 90 points. An assignment can be turned in from 4-7 days after the Submission Date for 75 points. Assignments turned in more than 7 days after the Submission Date will receive 0 points. You have the option to miss or drop one Written HW assignment.

Programming Homework
Programming homework will be assigned in conjunction with written homework. The due dates are listed above. Programs are expected to be written as Matlab m-file functions. In many cases, the instructor will provide the data to be analyzed, and the student is expected to turn in a computer generated solution along with a zip file containing the software. The instructor should be able to open the zip file, run the software successfully on his own machine, and obtain identical results. Any results must be written up and presented as a MS-Word or PDF-formatted report with appropriate explanation.

Each Programming Homework assignment can be turned in by the end of the day 11:59 pm of the Submission Date above for 100 points. An assignment can be turned in from 1-3 days after the Submission Date for 90 points. An assignment can be turned in from 4-7 days after the Submission Date for 75 points. Assignments turned in more than 7 days after the Submission Date will receive 0 points. You have the option to miss or drop one Programming HW assignment.

All programming HW must be emailed to

There will be one midterm exam on Oct 30, which will cover the material in the first half of the class. There will be no final exam. A missed exam cannot be made up unless there is a note from a doctor or the university.

Student Projects
Final Projects are required for Graduate Students. Students can choose to work individually or in groups of two to propose, perform, and present a final project for the course. This project will be a project that uses methods taught in this course to solve a data analysis or signal processing problem. Project proposals are written proposals describing the problem and proposed solution. The proposals are due on the date noted above, and must be approved by the instructor. The projects must be completed by Dec 14th and summarized in a project report due Dec 16th, which should follow the format of a short 4-8 page research paper including an abstract, introduction, method, results, conclusion, and references, along with the submission of a zip file containing the data and code. A poster session will be held during our Final Exam time, where the groups will present their work to others in a poster format as in a scientific meeting.

Undergraduate Grading
(students will choose by selecting whether to do a final project or an additional HW set)

Option I   Option II
Written HW35%  Written HW25%
Programming HW35%  Programming HW25%
Exams30%  Exams20%
Final Project:Optional Extra Credit  Final Project30%

Graduate Grading

Written HW25%
Programming HW25%
Final Project30%

Academic Integrity Every student has the responsibility to become familiar with the standards of academic integrity at the University. Faculty members must specify in their syllabi information about academic integrity, and may refer students to this policy for more information. Nonetheless, student claims of ignorance, unintentional error, or personal or academic pressures cannot be excuses for violation of academic integrity. Students are responsible for familiarizing themselves with the standards and behaving accordingly, and UAlbany faculty are responsible for teaching, modeling and upholding them. Anything less undermines the worth and value of our intellectual work, and the reputation and credibility of the University at Albany degree.
(University’s Standards of Academic Integrity Policy, Fall 2018)

The discovery of cheating on any exam or plagiarism on any homework will result in a failing grade and a report to the Chair of your department and/or the Dean of Undergraduate Studies.

Class Behavior While in class, students may not use cell phones or engage in any other type of disruptive behavior. Computers and tablets are allowed if being used for class activities, such as note-taking or visiting websites being discussed in class. The risk is being asked to leave the class. All students must be seated on time (that is, before the lecture begins); latecomers may be turned away at the door. Permission ahead of time is required for any student who must leave class early.