Audience was not aware of plan but there was one….Luke was going to use his Jedi mind trick but Jabba said that wasn’t going to work. A much easier thing would be for leia just to pay jabba but that would not make good cinema and make too much sense.

Fat shirtless guy must be rancor’s owner and the rancor probably was his pet.

They had no plan once they got to the pit. Luke and Han talk about it on the way there. Luke just says he’ll take care of it.