Hi,
I am also applying this season and don't have much more insight than you do on your profile. You should talk with your advisors/letter writers to establish a strategy if you want to reapply. Here are some comments anyway, but take them with a grain of salt.
(1) I am not sure. I think your profile is strongly affected by the not-so-good first years of undergrads. There are a lot good things in your profile too (research+clear progression at the end of your undergrad), but those first bad signals makes your profile "risky" from the point of view of ad. comittee. Given the number of applications they get, risky profiles can quickly be set aside.
(2) Unfortunately I don't think that there are simple things that you can do. You should ask yourself (1) what would be the ideal way to perfect your application (2) is it realistically feasible. Doing another year of RA can keep you occupied but I don't think it would add anything to your profile. Having another paper would certainly help, but since those are co-authored some ad. comittee could dismiss your contribution (this is something to talk about with your letter writers - make sure they highlight your contribution). A solo authored paper could probably help you a lot more, but I don't know if this possible (to be convincing this paper should probably be submitted by application deadlines). Taking more advanced math classes and getting extremely good grades (I mean being in the super top of the class) + maybe informal letters from math professors that you performed exceptionally would help make up for calculus (those could be attached to the letters sent by your main recommanders). I am think more Measure Theory or Advanced Probability Theory kind of classes (i.e. advanced proof-oriented classes). A math class with average grade will not help that much in my opinion.
As for (b), talk with your advisors but I am not sure that (american) masters performs noticely better that predocs. I would go with (a) not for the predoc but to have time to strenghten other aspects of the application. You could also try to get a better GRE score, I feel like stricly above of 167 can sometimes make a difference.
=> Overall I am rehashing advices that was given to me (I am also considering reapplying): the best way to up your profile is to get 2 to 3 unequivocally strong signals that can outshine the "risky" aspect of your profile. But I mean very strong signals that may not be realistic - you'll be the judge of that: exceptionnally good performance in advance math classes, solo authored working paper, etc.
Also, if you reapply you should ask your recommanders to explain more your somewhat poor performance in the beggining of undergrad. The circonstances you describe are completely understandable, but ad. com. are more likely to "understand" them if the explantations come from your letter writers.
(3) I don't think it is rude to ask, but don't except any positive response - they get so many applications they don't keep track why people are rejected.
Finally keep in my mind that this year is special. Even if you don't manage to up your profile, it can be worth it to try again next year - maybe trying some lower ranked schools. This is of course if this is not too costly for you to reapply, because your chances would only be marginally better (but this might be enough to get one offer).
Once again, I have no special authority to comment on your profile. I am just rehashing advices that was given to me, and maybe they don't apply.