I agree that math is your primary problem. Finance becomes highly mathematical at the grad level. They say that calc3, linear algebra, and stats is needed at a minimum. However, in all likelihhod you need more than this to stand a chance at top 20 schools.
In your first semester of a finance phd you'll prob prove black scholes in a class. This proof often uses Feynman-Kac, Ito, PDEs etc...not really trivial stuff and def not stuff you can pick up if you havent seen math past linear algebra.
I do disagree with the point about LACs tho. If
the OP attends a top LAC like Swarthmore/Amherst then this is generally viewed as being in the top 20 overall and they often sends students to PhD programs in every field.