Debates on wellbeing draw increasing evidence from studies on the use of time. Kahneman and Krueger investigate emotionally based utility and wellbeing, creating a composite ‘unhappiness’ (u-index) measure of the degree to which time-diary activities are perceived to be more unpleasant than pleasant. However, their analysis is primarily focussed activities rather than contextual factors – such as the co-presence of other people – in determining unpleasantness. This paper uses data from the US 2006 Princeton Affect and Time Survey (PATS) to examine the enjoyment of paid work and unpaid work in the context of present colleagues and family members. Results show that while paid work and unpaid work (domestic work and childcare) are consistently unpleasant activities for all people, paid work undertaken in the presence of colleagues and unpaid work in the presence of family are particularly unpleasant. Logistic models show that the odds of paid work with colleagues and unpaid work with family being unpleasant are generally greater than for other kinds of paid and unpaid work. In addition, women are no more likely to find paid work undertaken alone, with family or with friends unpleasant, while men are no more likely to find unpaid work undertaken alone, with colleagues or with friends unpleasant. Such unpleasantness is primarily due to high levels of stress and low levels of happiness while undertaking paid and unpaid work, though paid work with colleagues is also associated with more sadness, and unpaid work with less interest and more pain. Being in the workplace is considered unpleasant, regardless of activity and co-presence.