I think we are reading too much into what gorosei should have/could have done. While there's no basis for this, I assume the fruit hasn't awakened since the void century. By the way they talk it doesn't seem like they have been in their positions since the void century so there must be many...