Our defenses against infection rely on the ability of the immune system to distinguish invading pathogens from self. This task is exceptionally challenging, if not seemingly impossible, in the case of retroviruses that have integrated almost seamlessly into the host. This review examines the limits of innate and adaptive immune responses elicited by endogenous retroviruses and other retroelements, the targets of immune recognition, and the consequences for host health and disease. Contrary to theoretical expectation, endogenous retroelements retain substantial immunogenicity, which manifests most profoundly when their epigenetic repression is compromised, contributing to autoinflammatory and autoimmune disease and age-related inflammation. Nevertheless, recent evidence suggests that regulated immune reactivity to endogenous retroelements is integral to immune system development and function, underpinning cancer immunosurveillance, resistance to infection, and responses to the microbiota. Elucidation of the interaction points with endogenous retroelements will therefore deepen our understanding of immune system function and contribution to disease.