X's Algorithm: Transparent or Trapped?

X's recent publication of its 'for you' algorithm code, lauded by Elon Musk as a victory for transparency, faces scrutiny from researchers. Experts argue that the released code is a 'redacted' version, providing a superficial appearance of openness without enabling meaningful oversight or auditing.

The core innovation in the latest algorithm version is its reliance on a Grok-like large language model for ranking posts. This marks a significant shift from the previous iteration, which used hard-coded metrics like likes and shares. Now, the model's assessment of user engagement likelihood influences post ranking, increasing opacity.

Further complicating transparency efforts, X has omitted details previously shared in 2023 regarding the weighting of user interactions. Citing security reasons, the company has excluded data on how factors like replies or shares contribute to a post's visibility. This redaction hinders a complete understanding of the algorithm's mechanics.

Concerns also extend to the algorithm's training data, which remains undisclosed. Researchers like Mohsen Foroughifar from Carnegie Mellon University highlight that potential biases in this data could perpetuate discriminatory outcomes, even with internal model adjustments. The lack of access to this training data prevents thorough analysis and bias detection.

Comparisons are drawn to the broader AI landscape, as challenges in understanding social media algorithms are mirrored in generative AI. The opaqueness of X's system, with decision-making increasingly embedded in untrainable neural networks, reflects a larger trend of algorithmic complexity outpacing human comprehension, even for internal engineering teams.