I agree with that. But a lot of visual novels don't have an anime counterpart.

Would sticking them in "general anime" with a "Visual Novel:" tag really be organized enough? It might look a bit weird, but I think it is the best option.