Start by placing the room temperature butter in a mixing bowl. Use your electric mixer to beat the butter until it's fluffy and pale, about 2-3 minutes.
Next, add the creamy peanut butter to the bowl. Mix on medium speed until fully combined with the butter, scraping down the sides as necessary.
Pour in the milk gradually, mixing on low speed. Adjust the thickness by adding more milk if necessary.
Gradually add the powdered sugar while mixing on low speed. Once all the sugar is added, increase the speed and mix until smooth and creamy.
If the frosting is too thick, add a tiny bit more milk. If it's too thin, add a little more powdered sugar.
Taste your frosting and adjust by adding more peanut butter if desired.
