Yandex Practical ML Conf Showcases NEEDLE Search Benchmark and Ouroboros Self-Committing Coding Agent

At the Practical ML Conf conference held by Yandex on September 19, 2026, two talks in the 'Network' hall drew attention from the applied machine learning community: one proposing a new methodology for benchmarking search engines used in agentic pipelines, and another describing an autonomous coding agent that can modify and commit its own code.

NEEDLE: An Online Benchmark for Search APIs

The talk 'NEEDLE: an open benchmark for web search engines' was delivered by the author of the 'Love. Death. Transformers.' channel and was scheduled from 15:30 to 16:05 in the 'Network' hall. According to the source, NEEDLE is proposed as an online benchmark for search engines that uses "a continuous stream of fresh queries from news, articles, financial documents, court cases, and agent logs" rather than a static fixed set of queries.

The benchmark targets a gap identified in existing evaluation practice. As the source describes it, current benchmarks "test either the quality of language models or search models and algorithms, but not the quality of the final search engine — the API that an agent actually calls in a RAG pipeline." Static query sets, the source notes, are vulnerable to overfitting: they "can be found once, labeled, 'correct' documents can be cached, and ranking can be tuned to this list, after which the numbers stop reflecting live search quality."

Significant limitations apply at this stage. The source states that metrics, query selection procedure, and a description of the judge have not yet been published, meaning "the validity of the benchmark itself cannot be assessed, and it is premature to call it a ready-made tool for fair evaluation." Only the talk annotation, not the protocol, is available in open sources.

Ouroboros: A Coding Agent With the Right to Commit to Itself

Earlier in the same hall, from 13:20 to 13:55, Anton from the AbstractDL channel (@abstractDL) presented the talk 'Ouroboros: what's inside the agent that outperformed Codex, Claude Code, and Cursor'. According to the conference program annotation cited by the source, Ouroboros is "a working experiment, not a concept: a coding agent with a constitution, memory, and the right to commit to its own repository, which outperforms Codex, Claude Code, and Cursor on open benchmarks." The talk covered "the architecture, self-improvement loop, and pitfalls — from agents 'peeking' at answers to edits that nearly killed the system."

The source notes that the claimed performance advantages over competitors "have not yet been independently verified." It also flags a structural risk: "the combination of self-commits with evaluation on open benchmarks requires separate checking for overfitting to the same data," reflecting the classic Goodhart's law concern that edits may optimize scores on the same data against which the system is compared with competitors.

Access and What Remains Unknown

The 'Network' hall was broadcast live on the Yandex for ML YouTube channel, making both talks freely accessible without a conference ticket. The full conference program, spanning 'Data', 'Code', 'Server', and 'Network' halls, is available at pmlconf.yandex.ru/2026/program.

The source is explicit that both presentations are talks, not product releases: "there are no ready-to-implement tools today." The central open question following the conference is whether the NEEDLE protocol with its query stream and leaderboard, and the Ouroboros repository and evaluation protocol, will be published.