Sexual desire is typically higher in men than in women, with testosterone (T) thought to account for this difference as well as within-sex variation in desire in both women and men. However, few studies have incorporated both hormonal and social or psychological factors in studies of sexual desire. The present study addressed how three psychological domains (sexual-relational, stress-mood, body-embodiment) were related to links between T and sexual desire in healthy adults and whether dyadic and solitary desire showed associations with T. Participants (n = 196) were recruited as part of the Partnering, Physiology, and Health study, which had 105 men and 91 women who completed questionnaires and provided saliva for cortisol and T assays. T was positively linked to solitary desire in women, with masturbation frequency influencing this link. In contrast, T was negatively correlated with dyadic desire in women, but only when cortisol and perceived social stress were controlled. Replicating past findings, no significant correlations between T and desire in men were apparent, but these analyses showed that the null association remained even when psychological and confound variables were controlled. Men showed higher desire than women, but masturbation frequency rather than T influenced this difference. Results were discussed in terms of challenges to assumptions of clear links between T and desire, gendered approaches to T, and the unitarity of desire.