> ## Content Index > Fetch the complete content index at: https://www.tao.media/llms.txt > Use this file to discover other available public pages before exploring further. # ORO Launches ORO Bench With Daily Synthetic Shopping Environments on SN15 - URL: https://www.tao.media/oro-launches-oro-bench-with-daily-synthetic-shopping-environments-on-sn15/ - Published: 2026-09-23T21:48:33.000Z - Updated: 2026-09-23T21:48:33.000Z - Description: Bittensor Subnet 15 replaces static ShoppingBench evaluation with versioned EnvPacks, seven task families, and family-specific verifiers for agentic commerce. - Author: Bart Hillerich - Tags: Oro, Subnets, News [ORO](https://www.tao.media/tag/oro/) has launched [ORO Bench](https://oroagents.com/docs/oro-bench?ref=tao.media), a new evaluation system for shopping agents on Bittensor Subnet 15 that replaces a static benchmark loop with versioned, generated shopping environments. The team said in its [September 22 announcement](https://x.com/oroagents/status/2102475717367963815?ref=tao.media) that the system is "powered by a generator creating new synthetic shopping environments everyday." A companion [X Article](https://x.com/i/article/2102467536734523394?ref=tao.media) describes the benchmark as the current evaluation layer for SN15, where miners submit shopping agents and validators score them in sandboxed runs. The change moves ORO beyond its first ShoppingBench-based loop. Instead of evaluating agents against one fixed set of shopping problems, ORO Bench compiles task releases into immutable EnvPacks, gives agents dynamic tools inside generated environments, and scores performance with task-family-specific verifiers. Earlier this month, ORO became the first-ever crypto company with a live token to join Y Combinator. [Oro Becomes First Crypto Company to Join Y Combinator With Live TokenThe Fall 2026 acceptance puts a live-token Bittensor project inside YC as Oro builds open-source shopping agents and evaluation infrastructure.![](https://storage.ghost.io/c/78/0b/780ba906-b1a7-4bf0-873c-bdd5c32e5331/content/images/icon/Group-1321319358-5afefe9d-2ba3-41f9-9e71-97231247f50b.png)IntelligenceBart Hillerich![](https://storage.ghost.io/c/78/0b/780ba906-b1a7-4bf0-873c-bdd5c32e5331/content/images/thumbnail/ChatGPT-Image-Sep-8--2026--04_20_06-PM-079918d9-94e5-402a-998c-f94e8912f4db.png)](https://www.tao.media/oro-becomes-first-crypto-company-with-live-to/) ## Moving Beyond ShoppingBench ORO's first evaluation system used [ShoppingBench](https://ojs.aaai.org/index.php/AAAI/article/view/40640?ref=tao.media), a research benchmark for LLM-based shopping agents that involves a simulated shopping environment built around more than 2.5 million real-world products and user goals such as finding products, applying vouchers, managing budgets, and handling multi-product seller constraints. That structure gave SN15 an initial way to test agents in commerce-style tasks. But fixed benchmarks create a familiarity problem for competitive AI systems; once the task set, tools, and scoring pattern become stable enough, participants can optimize for the benchmark itself instead of for broader task-solving ability. As such, ORO Bench keeps the same broad objective, measuring agents that search, compare, rank, and justify shopping decisions, but it changes the evaluation substrate to keep performance improvements coming. ShoppingBench used static problem suites, a static product catalog, one shopping workflow, and fixed scoring. ORO Bench instead uses immutable EnvPacks with versioned task rosters, generated environment sessions across seven task families, dynamic tool schemas, family-specific verifiers, and trusted episode receipts. ## How EnvPacks Change SN15 Evaluation The core technical change is the EnvPack. An EnvPack is an immutable, content-addressed evaluation release. Each pack identifies the qualifying and race task rosters, environment contracts, verifier and reward contract versions, and supporting artifact identities needed to reproduce an evaluation. The pack is addressed by a SHA-256 identity, giving each evaluation a specific versioned reference rather than a loose pointer to whatever tasks happen to be active later. SN15's evaluation process includes both public qualifying work and hidden race work. A qualifying suite is bound to one immutable EnvPack, while a race is stamped with its EnvPack identity before race work is created. Each queued evaluation work item also freezes the same identity, so later suite changes cannot alter an in-flight run. Validators run a verification step before execution. They claim a run, download the required archive, verify its byte digest and parent pack identity, then validate the task roster, catalog references, and contract compatibility before sandboxed execution. Miners must write agents against the environment contract rather than a single hardcoded shopping workflow. Submitted agents export `agent_main(problem_data)`. In ORO Bench tasks, `problem_data["environment"]` gives the agent a public policy view with the goal, dynamic tool schemas, and episode limits, along with opaque binding fields that must be passed into environment calls. The agent then interacts with the environment through supplied tools until the environment reports completion, a limit is reached, or execution fails. ## Seven Task Families for Shopping Agents ORO Bench evaluates seven task families, each focused on a different commerce-agent capability: 1. Intent Decomposition — separating hard constraints from soft preferences. 2. Retrieval Recall — finding the right candidate set, including reformulation and pagination when results are thin. 3. Constraint Satisfaction — keeping shortlist items inside non-negotiable limits such as budget. 4. Preference Reasoning — translating fuzzy shopper requests into product attributes. 5. Ranking — ordering imperfect options in a way that matches a careful human decision. 6. Recovery — relaxing the right constraint when no clean answer exists. 7. Justification — explaining recommendations using product attributes a shopper can trust. The public labels are visible, but the hidden evaluation inputs and verifier authority are not. A release can carry a different task count or family mix, so agents read suite metadata instead of hardcoding assumptions about the evaluation. Real shopping tasks rarely fit a single pattern. A user may start with a strict budget, soften a brand preference, need multiple items from the same seller, or ask for a recommendation that requires evidence rather than only a product match. Splitting evaluation into task families lets ORO test those behaviors separately while still aggregating performance into competitive scores. ## Scoring Uses Trusted Episode Receipts ORO Bench scoring centers on trusted episode receipts instead of miner-reported outputs. In each task attempt, a trusted runtime starts a generated-environment session and exposes only the public policy view and opaque binding to the agent. When the episode ends, the runtime finalizes a receipt containing the task outcome, verifier verdict, reward components, step count, call trace, and public provenance. Each task family owns its verifier and reward calculation. Public results can show whether an episode was processed, whether the verifier accepted the outcome, how much paid reward the task contributed, counts of verifier checks, reward-component summaries, and the number of environment steps used. A completed episode can still receive zero reward. A validator run score is the mean paid reward across the expected task set, with agent failures counted as zero. Included validator runs are then aggregated into the agent's qualifying or race score. The receipt model keeps scoring tied to the evaluator-controlled environment. An agent can finish an interaction without satisfying the user's constraints, grounding its justification, or choosing a product that passes the verifier. ## What ORO Bench Changes for Bittensor SN15 ORO Bench turns shopping-agent competition on SN15 into a more dynamic evaluation market. Bittensor subnets depend on incentive design, where miners improve what validators reward. If the reward surface is too static, the competition can drift toward memorizing known cases or exploiting stable scoring patterns. ORO Bench counters that by separating public qualifying tasks from hidden race tasks and by using generated environments that can change across releases. During qualifying, agents are evaluated against the active public EnvPack suite. When qualifying closes, eligible challengers enter a race against a hidden roster from the same parent EnvPack release. Generated race scores and episode details remain withheld while the race is in flight and through the on-chain reveal embargo. That structure gives builders a public target for development while protecting the competitive set from direct tuning. It also makes SN15's benchmark more similar to real commerce work, where agents must handle new product sets, changing tool surfaces, ambiguous preferences, and constraints that are not identical to last week's examples.