It is perfectly fine to not go on a campus visit if you are accepted. There will be no negative consequences.

If you have not been accepted to a university then not attending a campus visit would have the same negative consequences as not making time for a job interview for any job you wanted to pursue.