Commission On Sales

The scenario you have cited is where A asks B to sell his house for $100 000 with a commissionof 5% which is $5000, and if the house is sold for more than $100 000 then B will charge a further commission of 60% on the amount exceeding $100 000. 

The ruling with regard to this is that when the commission or wage for a broker (agent) is fixed, such as when the agent says that I will charge $5000 or 5% of the selling price of the house when it is sold in return for the services offered to you; then this type of agreement is undoubtedly permissible.

As for the agent taking an extra percentage from the sale of an item as commission, as in your case a further 60% of the extra amount if it is sold for more than $100 000; the ruling now is as mentioned by Allamah Shaami Rahimahullah in his magnum opus, Radd al Muhtar that it is permitted to pay or take such a commission in the form of percentage from the sale of an item. Even though initially (prior to the sale) the exact commission in monetary value is unknown (as in your case when the agent will take an additional 60% of the amount exceeding $100 000), but as soon as the item is sold, it will be identified and fixed rendering the commission permissible. 

Also, given the current climate nowadays with regards to commission and dealing with brokers (agents). Allmah Shami Rahimahullah  states:

“Muhammed Ibn Salama was asked regarding the commission of a broker, so he said, “I hope there is no harm in it due to its widespread practice viz. Ta’amul, even though such a transaction should be invalid… however, the jurists permitted such transactions due to the people’s need in dealing with them.

Therefore, it is permissible to pay a commission to a broker, whether it is a fixed amount stipulated from before or an agreed upon percentage from the sale of the item.