Those groups have a crapton of members, so I think even among all the dross there's usually a couple of members who have
some modicum of charisma or talent. But I think in general the uniformity is the point. All of the groups have the same sound, they dress in very similar outfits, their songs have very few solo parts. If you're a fan in Japan you can feel comforted by the fact that the thing you like is going to keep doing the same exact thing over and over, and hundreds of thousands of other people are cheering along with you.
But these days Nogizaka is selling less than half of their peak and AKB less than a quarter of theirs. And some of the other groups are not all that far off from H!P territory. Part of what happened is the same phenomenon we saw with the OG MM. The first generations of members had a lot of space to carve out individual idiosyncracies and impress their uniqueness upon the public. But as more new members joined it they had to conform to the in-place machinery, and to their kouhai status, leaving less room for their personalities. So when the early generations graduate, they take fans with them and people aren't as attached to the blander younger generations. On top of that the format of the group starts to age as a concept and younger consumers don't find it as appealing. When H!P suffered its huge decline (can we still blame Charmy's hot mic?) management floundered for a few years but was able to reinvigorate things with new concepts. And as you point out, H!P has always had a stronger notion of individuality, usually giving every girl solo lines in almost every song, and allowing them to have less uniform...uniforms. And that has served them well.
Another big hit has been COVID. For a while at least, the very raison d'etre of Akimoto's groups was challenged as they had to severely dial back the fanservice and concertgoing experience (although some things had already changed since that horrible 2014 attack on Iriyama Anna and Kawaei Rina. Fortunately both members have
thrived since
then.)
Yet another thing has been the delayed-in-Japan but inexorable rise of streaming music. Nowadays for the biggest acts streams are arguably as important or even more, than physical sales. They relied historically on fans buying multiple versions of physical CDs for the hope of getting handshake tickets and the like. But people in general have becoming less concerned with buying CDs, and it shows up and down the charts.
And one more big change has been the rise of KPOP. It's overal sound is much more modern than the "sugar rush" of traditional idol pop, and it makes Akimoto's compositions sound dated. So the 48/46G ship has to eventually change course if it hopes to maintain its predominance.
But this old cruise liner has too much inertia to avert the looming iceberg. The vast majority of their back catalog consists of catchy mediocrities that are meant to be sung essentially in unison, like pop filler. And their current and future portends to be more of the same. But while they have to some extent crashed already, it's hard to feel sorry for them. They're still selling well into the 200K-700K range, and it certainly doesn't look like H!P will ever see those heights again. When our girls are putting out absolute bangers like Machigai Janai and still not cracking 100K, you have to wonder if Japan's eardrums have been permanently damaged by the steady diet of Johnny's/48/46G crap. Case in point, unlike Akimoto's other flagship groups, Hinatazaka46 has maintained a steady ~500K level of sales throughout the pandemic. But their
stuff still sounds identical to every other Akimoto group since 2005, and well, your question still stands.... "what am I missing?"
As I said at the outset, I think it ultimately boils down to people just liking their musical comfort food. Despite the rise of healthy cuisine, McD still vastly outsells every other chain restaurant with the exception of coffee house Starbucks, who themselves make mediocre coffee. Akimoto, I guess, is the
Ray Kroc of the Japanese music world.