Some remedies, like applying benzoyl peroxide or a warm compress, can help relieve hair bumps on your butt. Exfoliation can also help prevent ingrown hairs from forming. Share on Pinterest An ingrown ...