Gynecological and breast cancers are among the most prevalent malignancies in women. While survival has improved due to advances in treatment, many survivors experience challenges such as body image disturbances and reduced self-esteem. Physical activity has been identified as a potential protective factor, but evidence specific to this population remains limited. The aim of this study was to evaluate the differences in body image and self-esteem among physically active and inactive women with a history of gynecological and breast cancer, as compared to active and inactive healthy controls. A cross-sectional online survey was conducted in Poland. Participants were allocated to four groups according to cancer status and physical activity: inactive controls (CON-PI), active controls (CON-PA), inactive cancer survivors (CAN-PI), and active cancer survivors (CAN-PA). Physical activity was defined based on World Health Organization recommendations. Body image and self-esteem were assessed using the Body-Esteem Scale for Adolescents and Adults (BESAA) and the Rosenberg Self-Esteem Scale (RSES). Physically active women reported significantly higher BESAA and RSES scores compared to inactive women. The CAN-PI group consistently showed the lowest scores across all domains, while the CON-PA group reported the highest. Differences were particularly marked in appearance, attribution, and global self-esteem. Meeting WHO-recommended physical activity levels was associated with more positive body image and higher self-esteem in women after gynecological and breast cancer treatment. The findings highlight the potential role of physical activity in psychosocial well-being and support its inclusion in survivorship care.