Google's new flagship Gemini 4 faces internal criticism for prioritizing benchmark scores but underperforming in real-world testing.

The release of Gemini 4 has been met with internal skepticism; while its benchmark scores are impressive, its actual coding performance is inconsistent.

As reported by Wall Street News , Google officially released Gemini 4 Argon to the public on Wednesday, positioning it as the company's flagship artificial intelligence model developed over a long period of time.

However, according to Bloomberg, despite the model achieving leading scores in multiple benchmark tests, some Google insiders with direct access to the model have reservations about its actual performance, especially regarding coding capabilities.

Google explicitly denied the allegations, stating that claims about the Gemini 4's poor performance in areas such as encoding were "inaccurate," citing statements from Koray Kavukcuoglu, head of Google DeepMind, as evidence. Kavukcuoglu stated at a conference hosted by The Information last week:

I have full confidence in the team, and in my view, it's a certainty that we will always be at the forefront.

Facing competitive pressure from OpenAI and Anthropic's dual-track approach of advancing both models and products, the ability of Gemini 4 to deliver on its promises in real-world scenarios will directly impact Google's search, advertising, and cloud businesses. Following the news, Alphabet's after-hours gains narrowed significantly.

Internal disagreements: Strong benchmark results cannot mask actual shortcomings

According to Bloomberg, citing sources familiar with the matter, the Gemini 4 performed well in industry-standard benchmark tests, but its effectiveness was significantly reduced in actual use by employees, especially in coding tasks where it revealed obvious shortcomings.

The report quoted one of the sources as saying that the model lacks capabilities in the field of front-end design, which directly determines the visual presentation and interactive experience of applications and websites .

Some employees believe that Anthropic's Fable and OpenAI's Astra have surpassed Gemini in terms of capability iteration speed, and even if Gemini 4 performs at its best, it will still lag behind in several areas.

Another group of employees believes that the new model has caught up with the industry's top standards. A Google employee familiar with the model's development told Bloomberg that there is a "broad consensus" within the company that Gemini 4 is at the forefront and denies that it has any difficulties in real-world coding tasks.

Furthermore, reports citing sources familiar with the matter indicate that the Gemini 4 is massive. Large models typically mean higher inference costs, which will put additional pressure on the company's profit margins given Google's continued increase in capital expenditures.

"Over-optimization" or genuine ability?

Industry insiders attribute the above phenomenon to an industry problem known as "benchmaxxing"—that is, engineers focus more on getting high scores in standardized tests than on building truly usable products.

According to Bloomberg, two people familiar with Gemini 4 said the model appears to be affected by this issue. Edwin Chen, founder of AI startup Surge AI, was outspoken about it.

For example, it's like saying, 'My child scored very high on the SAT'—but SAT scores don't translate into real-world performance. This is an extremely harmful problem.

AI labs generally face this dilemma because clients often use benchmark scores as the main basis for evaluating the quality of models, which objectively drives labs to focus their efforts on "ranking up".

It's worth noting that Gemini 4 is not without its advantages. According to a source familiar with the matter, the model excels in multimodal tasks, such as extracting metadata from videos, and is also competitive in the fields of security and cybersecurity.

Google claims its model outperformed OpenAI's Astra model in a benchmark test measuring security skills. Furthermore, the Astra model can generate up to 1 million tokens in a single run, equivalent to approximately 750,000 English words.

A costly detour: The failure of the 3.5 Pro and the loss of talent.

Behind the release of Gemini 4 lies a costly detour in its research and development.

Google originally planned to launch the Gemini 3.5 Pro in June after the I/O conference in May, but the version was ultimately abandoned because it failed to meet internal targets.

Bloomberg analyst Mandeep Singh estimates that the cost of training a cutting-edge model in a single operation could reach as high as $400 million, and the actual expenditure would be even more substantial if the human resource costs of highly paid AI researchers were added.

At the same time, Google is also suffering from a continuous loss of talent. Several star AI researchers have left the company, including legendary engineer Jeff Dean, Nobel laureate John Jumper, and Noam Shazeer, who made significant contributions to the foundational technologies of the current AI wave.

In August of this year, Demis Hassabis, who had long been in charge of Google's AI research, became chairman, handing over the day-to-day operations of DeepMind to Kavukcuoglu.

Last November, Google released Gemini 3, which was widely praised and considered a significant turning point in the company's AI competitiveness. However, subsequent stagnation and setbacks have raised questions about its execution capabilities.

The competitive window is narrowing: competitors are closing in step by step.

The external pressures facing the Gemini 4 should not be underestimated.

Google's most direct risk lies in the fact that the Gemini series models form the technological foundation for core products such as Search, Maps, Gmail, and Chrome, each with a user base exceeding 1 billion.

Once a model falls behind in competitiveness, competitors will have more time to persuade developers, businesses, and consumers to choose their platform as the foundation for future search and software operations.

Meanwhile, OpenAI and Anthropic are accelerating their transformation from model providers to product companies, including advancing applications such as AI coding agents. Meta reportedly released its AI agent platform, Muse, earlier this month, which quickly topped the app download charts.

Google stated that despite the release of its previous Pro version in February of this year, the company's AI products have continued to grow, with the enterprise version of Gemini, the consumer chatbot application, and Google Search AI mode all expanding their user base, with the latter two now exceeding 1 billion users in total.

Google emphasizes that its massive user distribution system is a significant advantage over some of its competitors.

However, whether these advantages can translate into sustained leadership in the AI era still depends on whether the Gemini series can convince users in real-world application scenarios.

Risk Warning and DisclaimerInvesting involves risk; please exercise caution. This article does not constitute personal investment advice and does not take into account the specific investment objectives, financial situation, or needs of individual users. Users should consider whether any opinions, views, or conclusions in this article are suitable for their specific circumstances. Any investment decisions made based on this information are at your own risk.