nah. in north america it's not unusual for one's graduate degree to be in a somewhat different field from their undergrad. a friend of mine went from english literature to computer science at the university of waterloo. I know a girl who went from chemistry to control systems engineering at caltech. I am an electronics undergrad who got accepted into aerospace engineering grad programs at cornell and princeton (I mention school names only to demonstrate that it's not third-tier toilets accepting across fields out of some desperation, but the universities in the top 5-10.)
the point is, if you have a clear enough vision, going from chemistry into physics will not be a problem at all. besides, it's not like these fields are all *that* apart---both are natural sciences, and having a solid background in chemistry is a definite asset in many areas of physics. so yeah, go for it. a cogent statement of purpose is all you really need.
I believe the emphasis on `publications` on this board is misguided.