A high school student built a website that allows him to challenge AI models to a Minecraft construction

As the conventional comparative evaluation techniques of ai are inadequate, IA builders are resorting to more creative ways to evaluate the capacities of generative ai models. For a group of developers, that is Minecraft, the Sandbox construction game owned by Microsoft.

The website <a target="_blank" rel="nofollow" href="https://mcbench.ai/”>Minecraft reference point (or MC-Bench) developed in collaboration in ai models in boxes with each other in direct challenges to respond to the indications with Minecraft creations. Users can vote what model they did a better job, and only after voting can see what each Minecraft did.

For Adi Singh, the 12th grade student who began MC-Bench, the value of Minecraft is not so much the game itself, but the familiarity that people have with him, after all, is the best selling video game of all time. Even for people who have not played the game, it is still possible to evaluate which representation in a pineapple block is best.

“Minecraft allows people to see progress (of the development of ai) much more easily,” Singh told TechCrunch. “People are used to Minecraft, accustomed to appearance and environment.”

MC-BENCH currently lists eight people as voluntary taxpayers. Anthrope, Google, Openai and Alibaba have subsidized the use of their products by the project to manage reference indications, according to the MC-Bench website, but companies are not otherwise affiliated.

“We are currently making simple compilations to reflect on how far we have arrived from the GPT-3 era, but (we) we could see ourselves climb to these plans in a longer plans and tasks oriented to objectives,” said Singh. “The games could be a medium agent reasoning to prove that it is safer than in real life and more controllable for proof purposes, which makes it more ideal in my eyes.”

Other games like Pokémon Red, Street fighterand Pictionary have been used as experimental reference points for ai, partly because the art of the comparative evaluation of ai is notoriously complicated.

Researchers often try ai models in Standardized evaluationsBut many of these tests give a field advantage of origin. Due to the way they are trained, the models are naturally equipped in certain narrow types of problem solving, particularly the problem solving that requires memory or basic extrapolation.

In a nutshell, it is difficult to obtain what it means that Openai's GPT-4 can obtain in the 88th percentile in the LSAT, but cannot discern how many RS there is in the word “strawberry.” Anthropic Claude 3.7 sonnet He achieved an accuracy of 62.3% at a standardized software engineering point, but is worse to play Pokémon than most five -year -old children.

MC-BENCH is technically a programming reference point, since the models that write code are asked to create the requested construction, such as “Frosty The Snowman” or “a charming tropical beach cabin on a shore of virgin sand.”

But it is easier for most MC-Bench users to evaluate whether a snowman looks better than deepening the code, which gives the project a broader attraction and, therefore, the potential to collect more data on the models that constantly get best score.

If these scores are very equivalent to ai's utility, of course, it is in debate. However, Singh states that they are a strong signal.

“The current classification table is quite closely reflected with my own experience of using these models, which is different from many pure text reference points,” said Singh. “Maybe (MC-Bench) could be useful for companies to know if they are directed in the right direction.”