Since he became a tutor to Eris, I think he has stagnated on his own magic improvement. He focused on learning new languages, etc. and not honing his spells.
Also, there's a difference between casting spells and fighting using magic. I think he's lacking on the latter considerably, which is a bit of a surprise since he has been practicing swordsmanship so he has some level of instruction on fighting. But even so, point remains that he's still lacking on how to fight using magic besides whatever he has self-taught himself.
All those points aside, to your point... no one can cast faster than him that we know of (other than potentially the one that teleported and his ilk). So in a one on one setting, mages are probably at a disadvantage if even Rudeus is lacking.
But, I don't think it's an issue of him speeding up (though in time it may very well be, he's still growing), but more of an issue of accuracy and timing, and just overall battle awareness. His spell repertoire seems weak too, wonder if he'll learn new spells soon. He's honestly been half-assing it for a while now on this regard.
Anyways, loved the latest episode... a damn shame we have to wait for more. It's listed with 12 episodes, and that's not enough for the full story, so I hope we get more after. This is a series that deserves to been fully animated particularly since it's finished product as it is, and it's very popular to boot.