Assume that A, B , C, D ,E are the 5 people.
make a table for handshakes
Say, A, B, C are the 3 people who shake hands with exactly 3 people. Then,
A -> B, C, D
B -> A, C, D
C -> A, B, E (since D can not shake hands with 3 people)
Notice here that this table meets all the given requirements.
Now, count the handshakes
3 + 3 + 3 - 1 -1 -1
The subtractions are necessary since A shaking hands with B and vice versa is counted as 1 handshake not 2. Same goes for other handshakes too.
Hope this explains it.