Each side in a bidirectional RTP session assigns their source ports independently, i.e., there is no assumption that if Chunky sends audio to Raj on port 5000 (and RTCP on 5001), Chunky also has to receive audio on port 5000. (Imposing such a restriction on ports would make it difficult for a host to participate in several independent RTP sessions using different tools.)
Each side in a unicast session simply indicates to the other side where it wants to receive RTP packets, e.g., using SDP.
Note: that the SSRC values used for each source are always different.