In my experience, proving things like this, it's often easier to prove a stronger statement than a weaker one. Some of the techniques in my paper Factors of Sums of Powers of Binomial Coefficients might be relevant here.
It appears that if we let q be a root of the 5^k th cyclotomic polynomial, then the product of the first 5^k A's is very nice. This may imply your strong conjecture.
Indeed, if we let C be the product of the first 5^k A's, then it appears that C has entries
And of course, the result for 5^k follows from the result for 5, since the cyclotomic polynomial for 5^k is just the same as the one for 5 with x replaced by x^{5^{k-1}}.