Among gynecological tumors, ovarian cancer has the highest mortality rate and worst prognosis. Therefore, it is crucial to identify and develop novel methods and targets for treating ovarian cancer. Previous studies have shown that esculetin exerts antitumor effects in a variety of cancers, however, its anti-ovarian cancer effects and mechanisms of action remain unclear. In the present study, we investigated the anti-ovarian cancer effects and mechanisms of esculetin in A2780 and OVCAR3 ovarian cancer cells and established an xenograft ovarian cancer mouse model. Esculetin significantly inhibited ovarian cancer cell proliferation, blocked cell cycle progression, and promoted apoptosis and DNA damage in a concentration-dependent manner. Furthermore, esculetin inhibited tumor growth in an xenograft ovarian cancer mouse model. Moreover, RNA-seq showed that 2114 genes were significantly altered in A2780 cells after esculetin treatment. Kyoto Encyclopedia of Genes and Genomes pathway enrichment analysis indicated that these differentially expressed genes were mainly enriched in the calcium, HIF-1, and Rap1 signaling pathways. Interestingly, BNIP3 expression was notably upregulated in ovarian cancer cells after esculetin treatment. Finally, we found that low BNIP3 expression was correlated with poor prognosis in patients with ovarian cancer. These results prove that esculetin may be a valuable anti-ovarian cancer drug, and that BNIP3 is a potential treatment target for esculetin and a potential prognostic biomarker for ovarian cancer.