This question is asked a lot, and it is weird that Google Calendar doesn’t allow for this, given you can add alternate email addresses to your calendar.
If you want the RSVP to come from your alternate email then you need to respond directly from that email, not from the main or default email.