The incest taboo is more social than anything else, although it exists in some form in most human cultures studuied (research is ongoing as to why it seems so universal to frown on sibling/cousin pairin). Basically if there's something wrong with dad's (or mom's) genes and their grand kids mate, there's a better chance than random that some of the will inherit that same genetic problem. If there's nothing really wrong with with the grand parents (or there's something very desireable about the grand parents) then there's nothing scientifically 'bad' about inbreeding on a case by case basis. Problems tend to arise when a group aggressively inbreeds, as that group loses the genetic diversity nessecary to adapt well to changes.

On the flip side, if Grandpa or grandma had some awesome genes, perhaps excellent magic user genes.... then mating two grand children gives a better than random chance that a great-grandchild will inherit that gene...which sounds awesome if you have good genes...