Both schizophrenia (SCZ) and bipolar disorder (BD) are associated with disturbances in emotion processing. Previous studies suggest that patients with SCZ assess unpleasant pictures as less arousing than healthy controls (HC), while patients with BD assess neutral pictures as more arousing than HC. No previous studies have investigated whether there is a difference in emotional response across all three groups. Our aim was to explore whether there was a difference in the evaluation of valence and in arousal between SCZ, BD and HC for aversive and neutral pictures. We showed 72 pictures (neutral, non-socially aversive and socially aversive) from the International Affective Picture System (IAPS) to 347 subjects. There was a clear interaction effect between the diagnostic group and increasing picture aversiveness for both valence and arousal. There were no significant differences in valence ratings between the different groups or in arousal ratings on any type of stimuli between BD patients and HC. However, SCZ patients reported significantly lower arousal for aversive stimuli, particularly with a social content, when compared to BD patients and HC. This was more pronounced in females. The presence of lifetime psychotic symptoms did not influence emotional responses.