In this paper, we propose a novel framework for the analysis of task-related heart rate variability (HRV). Respiration and HRV are measured from 92 test participants while performing a chirp-breathing task consisting of breathing at a slowly increasing frequency under metronome guidance. A non-stationary stochastic model, belonging to the class of Locally Stationary Chirp Processes, is used to model the task-related HRV data, and its parameters are estimated with a novel inference method. The corresponding optimal mean-square error (MSE) time-frequency spectrum is derived and evaluated both with the individually estimated model parameters and the common process parameters. The results from the optimal spectrum are compared to the standard spectrogram with different window lengths and the Wigner-Ville spectrum, showing that the MSE optimal spectral estimator may be preferable to the other spectral estimates because of its optimal bias and variance properties. The estimated model parameters are considered as response variables in a regression analysis involving several physiological factors describing the test participants’ state of health, finding a correlation with gender, age, stress, and fitness. The proposed novel approach consisting of measuring HRV during a chirp-breathing task, a corresponding time-varying stochastic model, inference method, and optimal spectral estimator gives a complete framework for the study of task-related HRV in relation to factors describing both mental and physical health and may highlight otherwise overlooked correlations. This approach may be applied in general for the analysis of non-stationary data and especially in the case of task-related HRV, and it may be useful to search for physiological factors that determine individual differences.