One consequence of advocating for denial of rights based on sexual orientation may be that those who believe those rights shouldn't be denied may choose against inviting you.
You have every right to hold your beliefs, just like a conference has a right to choose against inviting you.