# Bayesian Data Analysis

Fall 2018

PHY/CSI/INF 451/551

Class Nos:5616, 5617, 5621, 5622, 6102, 6207

Lecture: TTh 1:15PM - 2:35PM, PH 225

Prof. Kevin H. Knuth

Physics Department

University at Albany

Albany NY USA

## News

There is an Extra Credit HW Assignment available for ALL students in the class. It consists of three problems. Each problem will be worth extra credit (despite the instructions in the old pdf, which I do not feel like rewriting at the moment). You may need to go through some of this material on multidimensional problems and covariance matrices. Each problem will be graded on a scale of 1-100pts. Those points will be added to your total HW points before computing the average HW score.

Matlab code from the Freely Falling Object lecture (Oct 18) has been posted online.

**UAlbany MATLAB License**: WikiPage with Instructions HERE

**Textbook**: Data Analysis: A Bayesian Tutorial by Sivia and Skilling, 2nd Edition

**Additional Texts (Recommended):**

Bayesian Probability Theory: Applications in the Physical Sciences

**Last Day to Drop** ('W' Assigned):

Graduate Students: Oct 23.

Undergraduate Students: Nov 6

## Readings

Note: Readings with links in (parentheses) are optional.

**Leisure**

Sep 6

AI researchers allege that machine learning is alchemy, Science, 2018 (link)

Frequentist vs Bayesian, Analytics India Magazine, 2018 (link)

Mike Lee Williams on Probabilistic Programming, ... PyMC3, InfoQ, 2018 (link)

**Assigned**

Sep 11

Cox, Probability, Frequency, and Reasonable Expectation C46

Knuth & Skilling, Foundations of Inference KS12

Skilling & Knuth, Measure, Probability, Quantum (SK18)

Sep 6

Wigner, The Unreasonable Effectiveness of Mathematics in the Natural Sciences, (W60)

Knuth, The Deeper Roles of Mathematics in Physical Laws, K15

Aug 28

Ch. 1, Sivia and Skilling

Jaynes, Bayesian Methods: General Background, J80

## Syllabus

Instructor: | Dr. Kevin H. Knuth, Associate Professor of Physics and Informatics |

Contact: | kknuth@albany.edu PH 211, 442-4501 |

Office Hours: | Thursday 2:45pm – 3:45pm, PH 211 |

TAs: | Bertrand Carado, Friday 10:00am - 11:00am, PH 228 |

Bertrand Carado, Friday 11:00am - 12:00pm, PH 224 | |

Siqian Zhao, Wednesday 10:00am - 11:00am, UAB434 | |

Required Text: | Data Analysis: A Bayesian Tutorial by Sivia and Skilling, 2nd Edition |

Required License: | MatLab Release 14 or Later: Student Edition |

Additional Texts (Recommended): | Bayesian Probability Theory: Applications in the Physical Sciences |

**Course Description:** Introduction to both the principles and practice of Bayesian and maximum entropy methods for data analysis, signal processing, and machine learning. This is a hands-on course that will introduce the use of the MATLAB computing language for software development. Students will learn to write their own Bayesian computer programs to solve problems relevant to physics, chemistry, biology, earth science, and signal processing, as well as hypothesis testing and error analysis. Optimization techniques to be covered include gradient ascent, fixed-point methods, and Markov chain Monte Carlo sampling techniques.

**3 credits.****Prerequisite(s): A MAT 214 (or equivalent) and A CSI 101 or A CSI 201.**

**Course Objectives:** Learn how to use the sum and product rules of probability to compute probabilities of various hypotheses. Learn how to use Bayes theorem to solve inference-based data analysis problems. Learn how to assign prior probabilities and likelihood functions based on the problem at hand. Learn both analytic and numerical techniques for computing the mean and mode of a probability density function as well as the accompanying uncertainties and the Bayesian evidence.

Week | Dates | Topics | Reading | HW |

1 | Aug 28 | Introduction | Ch. 1, J80 | |

Aug 30 | Problem Solving | |||

2 | Sep 4 | MATLAB Overview | HW1p:Sep 13 | |

Sep 6 | Foundations of Inference | (W60),K15 | ||

3 | Sep 11 | Foundations of Inference | C46,KS12, (GKS10) | |

Sep 13 | Bayes Theorem Examples | Ch. 2 | ||

4 | Sep 18 | Bayes Theorem Examples | HW2w:Sep 27 | |

Sep 20 | Bayes Theorem Examples | Ch. 3 | ||

5 | Sep 25 | Probability Density Funcs | ||

Sep 27 | Moments of Distributions | HW3p:Oct 16 | ||

6 | Oct 2 | Quantifying Uncertainty | Chs. 4 & 5 | HW4w:Oct 9 |

Oct 4 | Assigning Probs and Entropy | J68 Gif07 | ||

7 | Oct 9 | Marginalization | ||

Oct 11 | Ex: Length of a Pen | |||

8 | Oct 16 | Ex: Falling Object | ||

Oct 18 | Ex: Histogram Binning | K05a | ||

9 | Oct 23 | Review | ||

Oct 25 | Estimation in Many Dims | |||

10 | Oct 30 | MIDTERM | ||

Nov 1 | Ex: Source Separation | K99 | ||

11 | Nov 6 | Optimization Techniques | Ch 9,10 | |

Nov 8 | Sampling from PDFs | HW5p:Nov 26 | ||

12 | Nov 13 | Model Testing and Evidence | KHMMP15 | |

Nov 15 | Nested Sampling | |||

13 | Nov 20 | Nested Sampling | ||

Nov 22 | Thanksgiving | PROPOSALS NOV 27 | ||

14 | Nov 27 | Example: Exoplanet Characterization | PKA14 | HW6:Dec 12 |

Nov 29 | Example: Model Testing | |||

15 | Dec 4 | Metropolis Hastings MCMC | Ch. 7 | |

Dec 6 | Experimental Design | L03, KC10 | ||

16 | Dec 11 | EXTRA LECTURE | ||

Dec 14 | PROJECT POSTERS12pm-2pm in PH 225 |

**Website**

This course has a website that you can check for updates to the schedule or for special announcements.
http://knuthlab.rit.albany.edu/courses/2018/BayesianDataAnalysis/

**Written Homework**

Written homework assignments will be assigned approximately weekly. Typically, the solutions will be due 7 days after being assigned. All written homework assignments are expected to be completed and written in a neat and professional manner.

Each Written Homework assignment can be turned in by the end of the day 11:59 pm of the Submission Date above for 100 points. An assignment can be turned in from 1-3 days after the Submission Date for 90 points. An assignment can be turned in from 4-7 days after the Submission Date for 75 points. Assignments turned in more than 7 days after the Submission Date will receive 0 points. You have the option to miss or drop one Written HW assignment.

**Programming Homework**

Programming homework will be assigned in conjunction with written homework. The due dates are listed above. Programs are expected to be written as Matlab m-file functions. In many cases, the instructor will provide the data to be analyzed, and the student is expected to turn in a computer generated solution along with a zip file containing the software. The instructor should be able to open the zip file, run the software successfully on his own machine, and obtain identical results. **Any results must be written up and presented as a MS-Word or PDF-formatted report with appropriate explanation.**

Each Programming Homework assignment can be turned in by the end of the day 11:59 pm of the Submission Date above for 100 points. An assignment can be turned in from 1-3 days after the Submission Date for 90 points. An assignment can be turned in from 4-7 days after the Submission Date for 75 points. Assignments turned in more than 7 days after the Submission Date will receive 0 points. You have the option to miss or drop one Programming HW assignment.

**All programming HW must be emailed to knuthclass@gmail.com**

**Exams**

There will be one midterm exam on Oct 30, which will cover the material in the first half of the class. There will be no final exam. A missed exam cannot be made up unless there is a note from a doctor or the university.

**Student Projects**

Final Projects are required for Graduate Students. Students can choose to work individually or in groups of two to propose, perform, and present a final project for the course. This project will be a project that uses methods taught in this course to solve a data analysis or signal processing problem. Project proposals are written proposals describing the problem and proposed solution. The proposals are due on the date noted above, and must be approved by the instructor. The projects must be completed by Dec 14th and summarized in a project report due Dec 16th, which should follow the format of a short 4-8 page research paper including an abstract, introduction, method, results, conclusion, and references, along with the submission of a zip file containing the data and code. A poster session will be held during our Final Exam time, where the groups will present their work to others in a poster format as in a scientific meeting.

**Undergraduate Grading**

(students will choose by selecting whether to do a final project or an additional HW set)

Option I | Option II | ||||

Written HW | 35% | Written HW | 25% | ||

Programming HW | 35% | Programming HW | 25% | ||

Exams | 30% | Exams | 20% | ||

Final Project: | Optional Extra Credit | Final Project | 30% |

**Graduate Grading**

Written HW | 25% |

Programming HW | 25% |

Exams | 20% |

Final Project | 30% |

**Academic Integrity**
Every student has the responsibility to become familiar with the standards of academic integrity at the University. Faculty members must specify in their syllabi information about academic integrity, and may refer students to this policy for more information. Nonetheless, student claims of ignorance, unintentional error, or personal or academic pressures cannot be excuses for violation of academic integrity. Students are responsible for familiarizing themselves with the standards and behaving accordingly, and UAlbany faculty are responsible for teaching, modeling and upholding them. Anything less undermines the worth and value of our intellectual work, and the reputation and credibility of the University at Albany degree.

(University’s Standards of Academic Integrity Policy, Fall 2018)

http://www.albany.edu/undergraduate_bulletin/regulations.html

The discovery of cheating on any exam or plagiarism on any homework will result in a failing grade and a report to the Chair of your department and/or the Dean of Undergraduate Studies.

**Class Behavior**
While in class, students may not use cell phones or engage in any other type of disruptive behavior. Computers and tablets are allowed if being used for class activities, such as note-taking or visiting websites being discussed in class. The risk is being asked to leave the class. All students must be seated on time (that is, before the lecture begins); latecomers may be turned away at the door. Permission ahead of time is required for any student who must leave class early.