I just think it's needlessly confusing to have him both physically removing hearts, and then to visualize the soul swapping ability as switching hearts as well.
This again leads me to wonder why, if removing the heart has no adverse effect on someone, as Tashigi says, why did Smoker pass out in the first place when his was removed?
What exactly WAS the point of doing that to prevent them from escaping with the children...but then totally directing Luffy to them so he could help them escape?