USMA MA289: Mathematics of AI

Overview

This block emphasizes rigorous model comparison. Cadets move beyond asking whether a model fits the training data and instead ask whether an observed performance improvement is likely to generalize.

Neural networks are introduced in the preceding block on 20 October. This block builds on that foundation by emphasizing training behavior, regularization, tuning, and comparison with strong ensemble models. Review the 20 October neural-network introduction notes as needed.

Model Evaluation

• Train, validation, and test sets
• K-fold and stratified cross-validation
• Accuracy, precision, recall, and F1-score
• Confusion matrices
• ROC curves and AUC
• Regression error metrics
• Bias versus variance
• Hyperparameter tuning
• Public versus private leaderboard behavior

Ensemble Learning

• Bagging
• Random forests
• Gradient boosting
• Model averaging
• Variance reduction
• Hyperparameter tuning

Deep Learning

• Build on the 20 October introduction to neural networks
• Linear transformations and activation functions
• Single-layer and multilayer networks
• Loss functions
• Gradient descent and backpropagation
• Learning rate and regularization
• Dropout
• Comparison with conventional statistical-learning methods

Key Topics, Lesson Objectives, Readings, and Competition Work

Date	Key Topics	Lesson Objectives	Readings	Competition Work
27 Oct 2026	Competition 3 Launch Ensemble Modeling	Review the mathematical reason model averaging can reduce variance. Compare bagging, random forests, and boosting. Understand parallel versus sequential ensembles. Connect ensemble complexity to bias and variance. Related lesson notes: Decision Trees, Bagging & OOB Error .	ISLP Chapter 8: Tree-Based Methods §8.2.1 Bagging §8.2.2 Random Forests §8.2.3 Boosting §8.2.5 Summary of Tree Ensemble Methods Optional Python Lab: §8.3.3 Bagging and Random Forests §8.3.4 Boosting Géron Chapter 7: Ensemble Learning and Random Forests Voting Classifiers Bagging and Pasting Out-of-Bag Evaluation Random Forests Feature Importance AdaBoost Gradient Boosting Histogram-Based Gradient Boosting Stacking	Launch Competition 3. Establish a baseline model. Compare the baseline with at least one ensemble strategy. Record validation and Kaggle results.
03 Nov 2026	Model Evaluation & Hyperparameter Tuning	Design a tuning procedure that avoids test-set leakage. Compare cross-validation results rather than relying on one train/test split. Separate model selection from final model evaluation. Understand how hyperparameter search can itself overfit. Related lesson notes: Resampling & Cross-Validation | Model Selection & Regularization .	ISLP Chapter 5: Resampling Methods §5.1 Cross-Validation §5.2 The Bootstrap Chapter 6 §6.2.3 Selecting the Tuning Parameter Géron Chapter 1 Testing and Validating Chapter 2 Better Evaluation Using Cross-Validation Fine-Tune Your Model Grid Search Randomized Search Analyze the Best Models and Their Errors Evaluate the System on the Test Set	Tune ensemble hyperparameters. Maintain an experiment log. Compare cross-validation results with leaderboard results. Select candidate final models using local evidence.
10 Nov 2026	Neural Networks II: Training & Regularization	Understand the neural-network computational graph. Understand activation functions. Understand loss functions. Understand gradient descent and backpropagation. Understand regularization and dropout. Compare a neural network with conventional models. Review lesson notes: 20 October Neural Network Introduction | Gradient Descent .	ISLP Chapter 10: Deep Learning §10.1 Single Layer Neural Networks §10.2 Multilayer Neural Networks §10.6 When to Use Deep Learning §10.7 Fitting a Neural Network §10.7.1 Backpropagation §10.7.2 Regularization and Stochastic Gradient Descent §10.7.3 Dropout Learning §10.7.4 Network Tuning Géron Review Chapter 10: Introduction to Artificial Neural Networks with Keras The Perceptron The Multilayer Perceptron Backpropagation Activation Functions Regression MLPs Classification MLPs Building Neural Networks with Keras	When appropriate, train a small neural network. Compare neural-network validation performance with the strongest conventional model. Do not assume higher complexity produces better generalization.
17 Nov 2026	Competition 3 Review	Compare ensemble and neural-network strategies. Evaluate which improvements were robust across validation and competition data. Connect model complexity to generalization. Explain why the final model was selected. Related lesson notes: Neural Network Introduction | Cross-Validation .	ISLP Review Chapter 8: Ensemble Methods Review Chapter 10 §10.6 When to Use Deep Learning §10.7 Fitting a Neural Network §10.8 Interpolation and Double Descent Géron Chapter 11: Training Deep Neural Networks Better Activation Functions Faster Optimizers Learning Rate Scheduling Avoiding Overfitting Through Regularization L1 and L2 Regularization Dropout	Competition 3 Due 17 Nov 2026 NLT 2359 Submit final predictions and required competition materials.

Resources

• ISLP Chapter 5: Resampling Methods
• ISLP Chapter 8: Tree-Based Methods
• ISLP Chapter 10: Deep Learning
• Géron Chapter 2: End-to-End Machine Learning Project
• Géron Chapter 7: Ensemble Learning and Random Forests
• Géron Chapter 10: Introduction to Artificial Neural Networks with Keras
• Géron Chapter 11: Training Deep Neural Networks
• Kaggle Competition Hub
• Back to Home