I think that theory isn't silly.
If you noticed, his magic uses the same spells as the skulls (with different results though), during his infiltration to find the egg he had the same appearance.
He was recognised by a skull he smashed back in the Arch arc (as an exorcist or as a member of...