Jev-Omniは、文章、画像、音声、動画から、用意した選択肢を選ぶモデルです。質問と選択肢を渡すと、それぞれの確率を返します。

4種類の入力を同じモデルで扱えます。音声の内容を分類したり、画像に写った物を答えたり、メールに返信が必要かを判断したりできます。

音声10件、画像10件、架空の日本語メール10件、動画5件の計35件で試しました。そのうち画像・メール・動画の25件は、Clefとも比較しました。

Computer UseとBrowser Useの動画

Qwen3.8を主エージェントにして、decision modelを使った簡単なComputer Use(画面操作)とBrowser Use(ブラウザー操作)も試しました。Big Fiveの性格テストを題材にしています。

動画リンク: https://www.youtube.com/watch?v=Po3kZdTTMjw

結果の一覧

同じ入力と質問で、選択肢を元の順番と逆の順番で渡しました。並び方によって答えが変わるかを調べるためです。

「想定した答え」は、モデルの回答を見る前に、実験用の基準に沿って決めた答えです。表には、その答えと一致した数を示しています。

実験入力数Jev-OmniClef
E1:4種類の入力35元の順番33/35、逆順31/35対象外
E2:共通の画像・メール・動画25元の順番23/25、逆順21/25。E1の回答を再利用元の順番25/25、逆順25/25
E3:元の動画で緊急停止を質問5集計対象6回答のうち6件一致。4回答は除外同左
E4:接近する場面の追加動画5集計対象2回答のうち2件一致。8回答は除外同左

Jev-Omniは、元の順番で33/35(94.3%)、逆の順番で31/35(88.6%)が一致しました。画像I06とメールT01では、順番を変えると答えが変わりました。

この25件ではClefの成績が上でした。2回分を合わせると、Clefは50/50回答、Jev-Omniは44/50回答が一致しました。

動画の追加実験では、映像だけでは想定した答えを決められないものを、正答数の集計から外しました。

試し方

使用したモデル

モデル使用した版今回の役割
Jev-Omniakhilaaa3/Jev-Omni、revision c050d51354147985d13286cf4acf90f562f2c631音声、画像、文章、動画の選択肢を判定
Clefsimonlehmann/clef-NVFP4、revision 817ac58ad358f42489980ead62f8bf7cafc628c2画像、文章、動画でJev-Omniと比較

4種類の入力には、同じ読み込み済みのJev-Omniモデルを使いました。画像・音声・動画はGateway経由で渡し、文章は共通の分類処理を直接呼び出しました。

Clefは、Qwen3.8-27Bをもとに作られたモデルです。今回は、NVFP4とFP8を組み合わせた非公式の量子化版を使いました。量子化は、モデルの数値を小さい形式で保存する方法です。上流Qwenのrevisionは記録されていません。

Clefには、最初の実験と同じ画像・メール・動画を、同じ説明、質問、選択肢とともに渡しました。入力ファイルのSHA-256と指示を照合しています。Jev-Omniは、最初の実験で保存した回答を使いました。音声は比較していません。

選択肢の順番と確率

生成した入力が実験用の条件を満たすかを確認してから、モデルに渡しました。

順番を変えても、確率は選択肢の名前で対応づけて比べています。本文では小数点以下6桁に丸め、全桁は各回答のJSONに残しています。

確率はモデルが返した値です。判断が正しい確率として確かめたものではありません。

実験データの作り方

入力作成方法記録された主要な設定
音声ResembleAI/chatterbox-turbo英語、付属の標準音声、声の複製なし。mono 24 kHz PCM16 WAVを全長で使用
画像ByteDance/SDXL-Lightningsdxl_lightning_4step_unet.safetensors、4ステップ、guidance 0、1024×1024。生成PNGを品質90のJPEGに変換して評価
メール実験用の架空の日本語メールを直接作成生成モデルは使用していない
動画MiniMaxAI/MiniMax-H3とAlibaba PDD8記録されたAPI設定は9ステップ、flow shift 12、audio flow shift 3。短辺768、16:9、4秒を指定

PDD8で動画を生成しましたが、APIに指定したステップ数は9です。音声APIではseedが公開されていませんでした。画像・動画で指定したseedは生成記録にあります。

各入力のプロンプト、設定、作り直した記録は、生成記録のJSONと作成方法の説明にあります。

動画の読み込み方

評価には、生成した動画の先頭4秒を使いました。24 fps、96フレームのMP4です。生成元は107フレーム(4.458333秒)ですが、その末尾は使っていません。

モデル使用フレーム前処理
Jev-Omni16枚。0.0秒から3.75秒まで0.25秒間隔長辺を最大768ピクセルに調整
Clef8枚。source index [0,14,27,41,54,68,81,95]先頭・末尾を含む。事前resizeなしでdecodeし、Qwenの画像処理を適用

同じ動画ファイルでも、モデルが使うフレーム数と画像処理には違いがあります。処理時間は環境や計測範囲が異なり、各1回の記録しかないため、速さの比較には使っていません。

35件のデータで試しました

各種類10件、計40件を用意する予定でした。実験用の条件を満たしたのは、音声10件、画像10件、メール10件、動画5件です。用意できなかった動画5件は、不正解として数えていません。

入力採用数 / 予定数Jev-Omni:元の順番 / 逆順Clef:元の順番 / 逆順
英語の音声10/1010/10・10/10比較していない
物の画像10/1010/10・9/1010/10・10/10
日本語メール10/1010/10・9/1010/10・10/10
生成した動画5/103/5・3/55/5・5/5

音声10件:苦情と通常の問い合わせ

5件は苦情、5件は通常の問い合わせです。不具合を訴え、対応を求める内容をComplaintとしました。購入前の確認などはOrdinary inquiryとしました。丁寧な言い方でも、不具合への対応を求めていれば苦情に含めます。

Jev-Omniには実際のWAVを渡しました。下の原稿は音声を生成するために書いたもので、書き起こしではありません。

音声の内容は、別の音声認識処理をCPUで動かして確認しました。人が聴いて確認した記録はありません。A01のarrange delivery付近は認識が不確定で、A07・A09はWhisper base.enで確認し直しています。詳しくは音声の対応表にあります。

10件とも、選択肢の順番を変えても想定した答えと一致しました。

A01:配達済みなのに未着

A01:元のWAV全長、10.640秒。 元のファイル

想定した答え:Complaint

配送の失敗を訴え、配達か返金を求めています。

質問と選択肢

モデルに伝えた説明

  Assess only the audible customer message. A complaint expresses dissatisfaction with a prior service or product outcome, an unmet obligation or error, or seeks a remedy for a failure. Ordinary inquiries ask factual, prospective, or status questions without asserting failure. Politeness does not remove a complaint.
  

質問

  Is the audible customer message a complaint or an ordinary inquiry?
  

元の順番

  ["Complaint", "Ordinary inquiry"]
  

逆順

  ["Ordinary inquiry", "Complaint"]
  
モデル元の順番の回答逆順の回答想定回答との一致:元 / 逆
Jev-OmniComplaintComplaint一致 / 一致
選択肢Jev-Omni:元Jev-Omni:逆
Complaint0.8811040.999877
Ordinary inquiry0.1188960.000123
保存済み回答と実験ID
生成プロンプトと設定
  The tracking page says my parcel was delivered yesterday, but it never arrived. I checked the front door and asked my neighbours. Please find out where it went and arrange delivery or a refund.
  
  {
  "fixture_id": "A01",
  "cohorts": [
    "E1_JEV_INITIAL"
  ],
  "modality": "audio",
  "selected_attempt_id": "A01-attempt-01",
  "selected_attempt_index": 1,
  "evaluated_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/A01-attempt-01/asset.wav",
    "sha256": "e800911c146952b239ca3cf79df4ed73e40a2806fdb2a721eed19f69854a8865",
    "bytes": 510764
  },
  "native_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/A01-attempt-01/asset.wav",
    "sha256": "e800911c146952b239ca3cf79df4ed73e40a2806fdb2a721eed19f69854a8865",
    "bytes": 510764
  },
  "acceptance_receipt": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/A01-attempt-01/receipt.json",
    "sha256": "507ed6c07f101f640fe9e0f6731196b622c32ea7ce38e96dc448c46c27b68dde",
    "bytes": 2776
  },
  "observed_media": {
    "format": "PCM WAV",
    "channels": 1,
    "sample_rate_hz": 24000,
    "sample_width_bytes": 2,
    "duration_seconds": 10.64
  },
  "generator_id": "chatterbox_turbo",
  "prompt_field": "input",
  "seed": null,
  "seed_status": "Not exposed by the installed speech API; no seed or deterministic regeneration claim.",
  "submitted_settings": {
    "language": "en"
  },
  "fixture_or_configured_settings": {
    "language": "en",
    "voice": "bundled default",
    "voice_cloning": false,
    "format": "PCM WAV",
    "max_text_chars": 500,
    "max_duration_seconds": 30,
    "seed": null,
    "seed_availability": "Not exposed through installed speech API; no reproducibility claim."
  },
  "generation_request": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/A01-attempt-01/generation.request.json",
    "sha256": "ccdcdf5f85d9d2e35657d2db100f2453beefb6fce960f0f53835099aed6651f6",
    "bytes": 257
  },
  "recorded_runtime_identity_source": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/A01-attempt-01/generation.json",
    "sha256": "ce478961f08d999125ba2f9ea00dd00c1a2a3bf5a42ceac898b1d1324110e704",
    "bytes": 5426
  },
  "request_vs_configuration_notice": "Submitted settings and recorded runtime/fixture settings are separate evidence; a prepared generation record alone is not proof of execution. The selected asset and pre-inference acceptance receipt establish the retained input."
}
  

A02:二重請求の取り消し

A02:元のWAV全長、8.960秒。 元のファイル

想定した答え:Complaint

二重請求を訴え、余分な請求の取り消しを求めています。

質問と選択肢

モデルに伝えた説明

  Assess only the audible customer message. A complaint expresses dissatisfaction with a prior service or product outcome, an unmet obligation or error, or seeks a remedy for a failure. Ordinary inquiries ask factual, prospective, or status questions without asserting failure. Politeness does not remove a complaint.
  

質問

  Is the audible customer message a complaint or an ordinary inquiry?
  

元の順番

  ["Complaint", "Ordinary inquiry"]
  

逆順

  ["Ordinary inquiry", "Complaint"]
  
モデル元の順番の回答逆順の回答想定回答との一致:元 / 逆
Jev-OmniComplaintComplaint一致 / 一致
選択肢Jev-Omni:元Jev-Omni:逆
Complaint0.8661330.999871
Ordinary inquiry0.1338680.000129
保存済み回答と実験ID
生成プロンプトと設定
  I ordered one desk lamp, but my card was charged twice for the same order. The two payments have both posted. Could you please reverse the extra charge?
  
  {
  "fixture_id": "A02",
  "cohorts": [
    "E1_JEV_INITIAL"
  ],
  "modality": "audio",
  "selected_attempt_id": "A02-attempt-00",
  "selected_attempt_index": 0,
  "evaluated_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/A02-attempt-00/asset.wav",
    "sha256": "d008877d32667d9b9eb51bacaeb1eacb2ff7863fd337d5f9122b839cf94d0d71",
    "bytes": 430124
  },
  "native_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/A02-attempt-00/asset.wav",
    "sha256": "d008877d32667d9b9eb51bacaeb1eacb2ff7863fd337d5f9122b839cf94d0d71",
    "bytes": 430124
  },
  "acceptance_receipt": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/A02-attempt-00/receipt.json",
    "sha256": "afd31a9007a91d50903474c922276bb8a198de2a6ba18f93118e055b1ee0bfd4",
    "bytes": 1556
  },
  "observed_media": {
    "format": "PCM WAV",
    "channels": 1,
    "sample_rate_hz": 24000,
    "sample_width_bytes": 2,
    "duration_seconds": 8.96
  },
  "generator_id": "chatterbox_turbo",
  "prompt_field": "input",
  "seed": null,
  "seed_status": "Not exposed by the installed speech API; no seed or deterministic regeneration claim.",
  "submitted_settings": {
    "language": "en"
  },
  "fixture_or_configured_settings": {
    "language": "en",
    "voice": "bundled default",
    "voice_cloning": false,
    "format": "PCM WAV",
    "max_text_chars": 500,
    "max_duration_seconds": 30,
    "seed": null,
    "seed_availability": "Not exposed through installed speech API; no reproducibility claim."
  },
  "generation_request": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/A02-attempt-00/generation.request.json",
    "sha256": "e4aa97475b89879b93f0dfd8e191401d07362fa60de7e25823cc26a9626dc44f",
    "bytes": 216
  },
  "recorded_runtime_identity_source": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/A02-attempt-00/generation.json",
    "sha256": "bb5552a52f74fcb047a04b023b9f6d1aa94fda98996dbc45a97c98a0c90eec35",
    "bytes": 5385
  },
  "request_vs_configuration_notice": "Submitted settings and recorded runtime/fixture settings are separate evidence; a prepared generation record alone is not proof of execution. The selected asset and pre-inference acceptance receipt establish the retained input."
}
  

A03:壊れたケトルの交換

A03:元のWAV全長、7.320秒。 元のファイル

想定した答え:Complaint

届いた商品が壊れていたと報告し、交換を求めています。

質問と選択肢

モデルに伝えた説明

  Assess only the audible customer message. A complaint expresses dissatisfaction with a prior service or product outcome, an unmet obligation or error, or seeks a remedy for a failure. Ordinary inquiries ask factual, prospective, or status questions without asserting failure. Politeness does not remove a complaint.
  

質問

  Is the audible customer message a complaint or an ordinary inquiry?
  

元の順番

  ["Complaint", "Ordinary inquiry"]
  

逆順

  ["Ordinary inquiry", "Complaint"]
  
モデル元の順番の回答逆順の回答想定回答との一致:元 / 逆
Jev-OmniComplaintComplaint一致 / 一致
選択肢Jev-Omni:元Jev-Omni:逆
Complaint0.7592250.999659
Ordinary inquiry0.2407750.000341
保存済み回答と実験ID
生成プロンプトと設定
  The kettle arrived this morning with a cracked handle, so I cannot use it safely. Please tell me how to get a replacement for the damaged one.
  
  {
  "fixture_id": "A03",
  "cohorts": [
    "E1_JEV_INITIAL"
  ],
  "modality": "audio",
  "selected_attempt_id": "A03-attempt-00",
  "selected_attempt_index": 0,
  "evaluated_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/A03-attempt-00/asset.wav",
    "sha256": "f67d95c676f356236d4bf8bb642ca4c7040e1bc0eeac6f7d4af36881b38080b5",
    "bytes": 351404
  },
  "native_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/A03-attempt-00/asset.wav",
    "sha256": "f67d95c676f356236d4bf8bb642ca4c7040e1bc0eeac6f7d4af36881b38080b5",
    "bytes": 351404
  },
  "acceptance_receipt": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/A03-attempt-00/receipt.json",
    "sha256": "7b228f2ec3c6ddc09adc6a5f47cfc24df7bc3f872bb902343c1a52a6b2ab4439",
    "bytes": 1529
  },
  "observed_media": {
    "format": "PCM WAV",
    "channels": 1,
    "sample_rate_hz": 24000,
    "sample_width_bytes": 2,
    "duration_seconds": 7.32
  },
  "generator_id": "chatterbox_turbo",
  "prompt_field": "input",
  "seed": null,
  "seed_status": "Not exposed by the installed speech API; no seed or deterministic regeneration claim.",
  "submitted_settings": {
    "language": "en"
  },
  "fixture_or_configured_settings": {
    "language": "en",
    "voice": "bundled default",
    "voice_cloning": false,
    "format": "PCM WAV",
    "max_text_chars": 500,
    "max_duration_seconds": 30,
    "seed": null,
    "seed_availability": "Not exposed through installed speech API; no reproducibility claim."
  },
  "generation_request": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/A03-attempt-00/generation.request.json",
    "sha256": "351d1b49bee8ffb41e49b70baf60c1c209b65120f0ffcd2b9dda14c9c6c9f3a9",
    "bytes": 206
  },
  "recorded_runtime_identity_source": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/A03-attempt-00/generation.json",
    "sha256": "5c14507ae0bd331d961bb9cd89f0e4d1fef308f0e4820dc6988068e2147caf4f",
    "bytes": 5375
  },
  "request_vs_configuration_notice": "Submitted settings and recorded runtime/fixture settings are separate evidence; a prepared generation record alone is not proof of execution. The selected asset and pre-inference acceptance receipt establish the retained input."
}
  

A04:約束した折り返し電話がない

A04:元のWAV全長、9.600秒。 元のファイル

想定した答え:Complaint

丁寧な言い方ですが、約束した折り返し電話がないことへの対応を求めています。

質問と選択肢

モデルに伝えた説明

  Assess only the audible customer message. A complaint expresses dissatisfaction with a prior service or product outcome, an unmet obligation or error, or seeks a remedy for a failure. Ordinary inquiries ask factual, prospective, or status questions without asserting failure. Politeness does not remove a complaint.
  

質問

  Is the audible customer message a complaint or an ordinary inquiry?
  

元の順番

  ["Complaint", "Ordinary inquiry"]
  

逆順

  ["Ordinary inquiry", "Complaint"]
  
モデル元の順番の回答逆順の回答想定回答との一致:元 / 逆
Jev-OmniComplaintComplaint一致 / 一致
選択肢Jev-Omni:元Jev-Omni:逆
Complaint0.8261510.999812
Ordinary inquiry0.1738490.000188
保存済み回答と実験ID
生成プロンプトと設定
  Hello, I know your team is busy. You said someone would call me back by Tuesday, and it is Thursday now. I have not heard from anyone. Could you please arrange that callback?
  
  {
  "fixture_id": "A04",
  "cohorts": [
    "E1_JEV_INITIAL"
  ],
  "modality": "audio",
  "selected_attempt_id": "A04-attempt-00",
  "selected_attempt_index": 0,
  "evaluated_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/A04-attempt-00/asset.wav",
    "sha256": "6f7d5f65db14fbf9b88268e634b49757c76b61f358fc8796e5fcd830e651fdd7",
    "bytes": 460844
  },
  "native_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/A04-attempt-00/asset.wav",
    "sha256": "6f7d5f65db14fbf9b88268e634b49757c76b61f358fc8796e5fcd830e651fdd7",
    "bytes": 460844
  },
  "acceptance_receipt": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/A04-attempt-00/receipt.json",
    "sha256": "999b90ab815a04a3b453eb1a8ef6451ca4f401fae6115801309d7ae971d7130f",
    "bytes": 1605
  },
  "observed_media": {
    "format": "PCM WAV",
    "channels": 1,
    "sample_rate_hz": 24000,
    "sample_width_bytes": 2,
    "duration_seconds": 9.6
  },
  "generator_id": "chatterbox_turbo",
  "prompt_field": "input",
  "seed": null,
  "seed_status": "Not exposed by the installed speech API; no seed or deterministic regeneration claim.",
  "submitted_settings": {
    "language": "en"
  },
  "fixture_or_configured_settings": {
    "language": "en",
    "voice": "bundled default",
    "voice_cloning": false,
    "format": "PCM WAV",
    "max_text_chars": 500,
    "max_duration_seconds": 30,
    "seed": null,
    "seed_availability": "Not exposed through installed speech API; no reproducibility claim."
  },
  "generation_request": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/A04-attempt-00/generation.request.json",
    "sha256": "da956535de808091ebfdffb4cf158914701162cb82d9fdc9e6329fc06176b74d",
    "bytes": 238
  },
  "recorded_runtime_identity_source": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/A04-attempt-00/generation.json",
    "sha256": "58db357073faca8ba978fe23c40786744452c996bda4f373936f83bdde99e4d6",
    "bytes": 5407
  },
  "request_vs_configuration_notice": "Submitted settings and recorded runtime/fixture settings are separate evidence; a prepared generation record alone is not proof of execution. The selected asset and pre-inference acceptance receipt establish the retained input."
}
  

A05:ノートの紙への不満

A05:元のWAV全長、9.280秒。 元のファイル

想定した答え:Complaint

使った商品の紙質に不満があり、対応を求めています。

質問と選択肢

モデルに伝えた説明

  Assess only the audible customer message. A complaint expresses dissatisfaction with a prior service or product outcome, an unmet obligation or error, or seeks a remedy for a failure. Ordinary inquiries ask factual, prospective, or status questions without asserting failure. Politeness does not remove a complaint.
  

質問

  Is the audible customer message a complaint or an ordinary inquiry?
  

元の順番

  ["Complaint", "Ordinary inquiry"]
  

逆順

  ["Ordinary inquiry", "Complaint"]
  
モデル元の順番の回答逆順の回答想定回答との一致:元 / 逆
Jev-OmniComplaintComplaint一致 / 一致
選択肢Jev-Omni:元Jev-Omni:逆
Complaint0.9033790.999925
Ordinary inquiry0.0966210.000075
保存済み回答と実験ID
生成プロンプトと設定
  I tried the new notebook this week. The pages are thinner than I expected, and ink shows through when I write. It is a little disappointing. Is there anything you can do about it?
  
  {
  "fixture_id": "A05",
  "cohorts": [
    "E1_JEV_INITIAL"
  ],
  "modality": "audio",
  "selected_attempt_id": "A05-attempt-00",
  "selected_attempt_index": 0,
  "evaluated_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/A05-attempt-00/asset.wav",
    "sha256": "c0d801a429dfd9312491b4be3c2423647045c2a6be03c8094fbea48faa45e463",
    "bytes": 445484
  },
  "native_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/A05-attempt-00/asset.wav",
    "sha256": "c0d801a429dfd9312491b4be3c2423647045c2a6be03c8094fbea48faa45e463",
    "bytes": 445484
  },
  "acceptance_receipt": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/A05-attempt-00/receipt.json",
    "sha256": "c91492a815d9370c66c7a6c88685d09104f067a97d22149cbfe62044426d0124",
    "bytes": 1615
  },
  "observed_media": {
    "format": "PCM WAV",
    "channels": 1,
    "sample_rate_hz": 24000,
    "sample_width_bytes": 2,
    "duration_seconds": 9.28
  },
  "generator_id": "chatterbox_turbo",
  "prompt_field": "input",
  "seed": null,
  "seed_status": "Not exposed by the installed speech API; no seed or deterministic regeneration claim.",
  "submitted_settings": {
    "language": "en"
  },
  "fixture_or_configured_settings": {
    "language": "en",
    "voice": "bundled default",
    "voice_cloning": false,
    "format": "PCM WAV",
    "max_text_chars": 500,
    "max_duration_seconds": 30,
    "seed": null,
    "seed_availability": "Not exposed through installed speech API; no reproducibility claim."
  },
  "generation_request": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/A05-attempt-00/generation.request.json",
    "sha256": "29e760e36441683f98fe17f0283186931c6413930c83ff6fe3577f60ad60f3a7",
    "bytes": 243
  },
  "recorded_runtime_identity_source": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/A05-attempt-00/generation.json",
    "sha256": "e8344046caf8278b5e6a0e04e41bd553cc6df993c17fe11656cd25bccd138b02",
    "bytes": 5412
  },
  "request_vs_configuration_notice": "Submitted settings and recorded runtime/fixture settings are separate evidence; a prepared generation record alone is not proof of execution. The selected asset and pre-inference acceptance receipt establish the retained input."
}
  

A06:土曜日の開店時間

A06:元のWAV全長、5.240秒。 元のファイル

想定した答え:Ordinary inquiry

不具合は訴えず、営業時間を尋ねています。

質問と選択肢

モデルに伝えた説明

  Assess only the audible customer message. A complaint expresses dissatisfaction with a prior service or product outcome, an unmet obligation or error, or seeks a remedy for a failure. Ordinary inquiries ask factual, prospective, or status questions without asserting failure. Politeness does not remove a complaint.
  

質問

  Is the audible customer message a complaint or an ordinary inquiry?
  

元の順番

  ["Complaint", "Ordinary inquiry"]
  

逆順

  ["Ordinary inquiry", "Complaint"]
  
モデル元の順番の回答逆順の回答想定回答との一致:元 / 逆
Jev-OmniOrdinary inquiryOrdinary inquiry一致 / 一致
選択肢Jev-Omni:元Jev-Omni:逆
Complaint0.0002530.018405
Ordinary inquiry0.9997470.981595
保存済み回答と実験ID
生成プロンプトと設定
  Hi, what time does your shop open on Saturdays? I would like to visit before lunch.
  
  {
  "fixture_id": "A06",
  "cohorts": [
    "E1_JEV_INITIAL"
  ],
  "modality": "audio",
  "selected_attempt_id": "A06-attempt-00",
  "selected_attempt_index": 0,
  "evaluated_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/A06-attempt-00/asset.wav",
    "sha256": "6e237b86d8c3f23de45baa7b0d03875983515fca29e9844d6710f87a2b976607",
    "bytes": 251564
  },
  "native_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/A06-attempt-00/asset.wav",
    "sha256": "6e237b86d8c3f23de45baa7b0d03875983515fca29e9844d6710f87a2b976607",
    "bytes": 251564
  },
  "acceptance_receipt": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/A06-attempt-00/receipt.json",
    "sha256": "4971da0cbd6bc846d5e5c9139a470fade40ff9b204e3e80c79c2f673a4b4da72",
    "bytes": 1466
  },
  "observed_media": {
    "format": "PCM WAV",
    "channels": 1,
    "sample_rate_hz": 24000,
    "sample_width_bytes": 2,
    "duration_seconds": 5.24
  },
  "generator_id": "chatterbox_turbo",
  "prompt_field": "input",
  "seed": null,
  "seed_status": "Not exposed by the installed speech API; no seed or deterministic regeneration claim.",
  "submitted_settings": {
    "language": "en"
  },
  "fixture_or_configured_settings": {
    "language": "en",
    "voice": "bundled default",
    "voice_cloning": false,
    "format": "PCM WAV",
    "max_text_chars": 500,
    "max_duration_seconds": 30,
    "seed": null,
    "seed_availability": "Not exposed through installed speech API; no reproducibility claim."
  },
  "generation_request": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/A06-attempt-00/generation.request.json",
    "sha256": "dc6150fb5c906e0fcd77bcc8e3f9439fe5f39c8980fed749b27de6e8b53fd1f3",
    "bytes": 147
  },
  "recorded_runtime_identity_source": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/A06-attempt-00/generation.json",
    "sha256": "b79c742a5de60582587b9e23e2bc2874d025dc40c3d866d71560e57bf31cc189",
    "bytes": 5316
  },
  "request_vs_configuration_notice": "Submitted settings and recorded runtime/fixture settings are separate evidence; a prepared generation record alone is not proof of execution. The selected asset and pre-inference acceptance receipt establish the retained input."
}
  

A07:購入前の配送時期の確認

A07:元のWAV全長、7.960秒。 元のファイル

想定した答え:Ordinary inquiry

購入前に、配送時期を尋ねています。

質問と選択肢

モデルに伝えた説明

  Assess only the audible customer message. A complaint expresses dissatisfaction with a prior service or product outcome, an unmet obligation or error, or seeks a remedy for a failure. Ordinary inquiries ask factual, prospective, or status questions without asserting failure. Politeness does not remove a complaint.
  

質問

  Is the audible customer message a complaint or an ordinary inquiry?
  

元の順番

  ["Complaint", "Ordinary inquiry"]
  

逆順

  ["Ordinary inquiry", "Complaint"]
  
モデル元の順番の回答逆順の回答想定回答との一致:元 / 逆
Jev-OmniOrdinary inquiryOrdinary inquiry一致 / 一致
選択肢Jev-Omni:元Jev-Omni:逆
Complaint0.0001970.026156
Ordinary inquiry0.9998030.973844
保存済み回答と実験ID
生成プロンプトと設定
  If I place an order for the blue chair today, when would you expect it to arrive? I have not ordered yet and am planning a room update.
  
  {
  "fixture_id": "A07",
  "cohorts": [
    "E1_JEV_INITIAL"
  ],
  "modality": "audio",
  "selected_attempt_id": "A07-attempt-00",
  "selected_attempt_index": 0,
  "evaluated_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/A07-attempt-00/asset.wav",
    "sha256": "2e4f53fe892d39b56c291904430db5e920cf35c37f45617b643840014f121735",
    "bytes": 382124
  },
  "native_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/A07-attempt-00/asset.wav",
    "sha256": "2e4f53fe892d39b56c291904430db5e920cf35c37f45617b643840014f121735",
    "bytes": 382124
  },
  "acceptance_receipt": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/A07-attempt-00/receipt.json",
    "sha256": "ada91499e1f86c9478006cfe9ec3777769f90035b9d69ce6d1fdfa2dccf95bc4",
    "bytes": 2280
  },
  "observed_media": {
    "format": "PCM WAV",
    "channels": 1,
    "sample_rate_hz": 24000,
    "sample_width_bytes": 2,
    "duration_seconds": 7.96
  },
  "generator_id": "chatterbox_turbo",
  "prompt_field": "input",
  "seed": null,
  "seed_status": "Not exposed by the installed speech API; no seed or deterministic regeneration claim.",
  "submitted_settings": {
    "language": "en"
  },
  "fixture_or_configured_settings": {
    "language": "en",
    "voice": "bundled default",
    "voice_cloning": false,
    "format": "PCM WAV",
    "max_text_chars": 500,
    "max_duration_seconds": 30,
    "seed": null,
    "seed_availability": "Not exposed through installed speech API; no reproducibility claim."
  },
  "generation_request": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/A07-attempt-00/generation.request.json",
    "sha256": "ff038681f92f9c842a7403ce9772f6b4cb3b99d64e4c8d717f8e4b2f107982f9",
    "bytes": 199
  },
  "recorded_runtime_identity_source": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/A07-attempt-00/generation.json",
    "sha256": "5df713a744a58368b1a4d8b9132ab1aeeb197fac4bb0c34ffebbc62aed91cda5",
    "bytes": 5368
  },
  "request_vs_configuration_notice": "Submitted settings and recorded runtime/fixture settings are separate evidence; a prepared generation record alone is not proof of execution. The selected asset and pre-inference acceptance receipt establish the retained input."
}
  

A08:購入前の返品条件の確認

A08:元のWAV全長、8.120秒。 元のファイル

想定した答え:Ordinary inquiry

購入前に返品条件を尋ねています。商品やサービスの不具合は訴えていません。

質問と選択肢

モデルに伝えた説明

  Assess only the audible customer message. A complaint expresses dissatisfaction with a prior service or product outcome, an unmet obligation or error, or seeks a remedy for a failure. Ordinary inquiries ask factual, prospective, or status questions without asserting failure. Politeness does not remove a complaint.
  

質問

  Is the audible customer message a complaint or an ordinary inquiry?
  

元の順番

  ["Complaint", "Ordinary inquiry"]
  

逆順

  ["Ordinary inquiry", "Complaint"]
  
モデル元の順番の回答逆順の回答想定回答との一致:元 / 逆
Jev-OmniOrdinary inquiryOrdinary inquiry一致 / 一致
選択肢Jev-Omni:元Jev-Omni:逆
Complaint0.0004040.054218
Ordinary inquiry0.9995960.945782
保存済み回答と実験ID
生成プロンプトと設定
  Could you explain your return policy for an unopened item? I am comparing gift options and would like to know the return window before buying.
  
  {
  "fixture_id": "A08",
  "cohorts": [
    "E1_JEV_INITIAL"
  ],
  "modality": "audio",
  "selected_attempt_id": "A08-attempt-00",
  "selected_attempt_index": 0,
  "evaluated_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/A08-attempt-00/asset.wav",
    "sha256": "1cce8b3404b77b4da2ef342166209f8293ed97cf2bc931e13e430029f309a1f0",
    "bytes": 389804
  },
  "native_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/A08-attempt-00/asset.wav",
    "sha256": "1cce8b3404b77b4da2ef342166209f8293ed97cf2bc931e13e430029f309a1f0",
    "bytes": 389804
  },
  "acceptance_receipt": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/A08-attempt-00/receipt.json",
    "sha256": "6c4354f781fe2a6c47dd5df69824132b19d70a14737ba92c76e55f846252e67c",
    "bytes": 1561
  },
  "observed_media": {
    "format": "PCM WAV",
    "channels": 1,
    "sample_rate_hz": 24000,
    "sample_width_bytes": 2,
    "duration_seconds": 8.12
  },
  "generator_id": "chatterbox_turbo",
  "prompt_field": "input",
  "seed": null,
  "seed_status": "Not exposed by the installed speech API; no seed or deterministic regeneration claim.",
  "submitted_settings": {
    "language": "en"
  },
  "fixture_or_configured_settings": {
    "language": "en",
    "voice": "bundled default",
    "voice_cloning": false,
    "format": "PCM WAV",
    "max_text_chars": 500,
    "max_duration_seconds": 30,
    "seed": null,
    "seed_availability": "Not exposed through installed speech API; no reproducibility claim."
  },
  "generation_request": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/A08-attempt-00/generation.request.json",
    "sha256": "d11f045befccd65cbdc20dc971a63a143717c615b6e6be0a7db5da7f91dbf852",
    "bytes": 206
  },
  "recorded_runtime_identity_source": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/A08-attempt-00/generation.json",
    "sha256": "aa2f468e032a756233827f702b76185cd8a869473239cfac70e745ed27b5d5bb",
    "bytes": 5375
  },
  "request_vs_configuration_notice": "Submitted settings and recorded runtime/fixture settings are separate evidence; a prepared generation record alone is not proof of execution. The selected asset and pre-inference acceptance receipt establish the retained input."
}
  

A09:期限内の注文の状況確認

A09:元のWAV全長、8.760秒。 元のファイル

想定した答え:Ordinary inquiry

約束した期限の前に、注文の状況を尋ねています。遅延は訴えていません。

質問と選択肢

モデルに伝えた説明

  Assess only the audible customer message. A complaint expresses dissatisfaction with a prior service or product outcome, an unmet obligation or error, or seeks a remedy for a failure. Ordinary inquiries ask factual, prospective, or status questions without asserting failure. Politeness does not remove a complaint.
  

質問

  Is the audible customer message a complaint or an ordinary inquiry?
  

元の順番

  ["Complaint", "Ordinary inquiry"]
  

逆順

  ["Ordinary inquiry", "Complaint"]
  
モデル元の順番の回答逆順の回答想定回答との一致:元 / 逆
Jev-OmniOrdinary inquiryOrdinary inquiry一致 / 一致
選択肢Jev-Omni:元Jev-Omni:逆
Complaint0.0004960.072014
Ordinary inquiry0.9995040.927986
保存済み回答と実験ID
生成プロンプトと設定
  My order is due to arrive next Friday, so it is still within the promised time. Could you tell me whether it has been packed yet? I just want to plan when to be home.
  
  {
  "fixture_id": "A09",
  "cohorts": [
    "E1_JEV_INITIAL"
  ],
  "modality": "audio",
  "selected_attempt_id": "A09-attempt-00",
  "selected_attempt_index": 0,
  "evaluated_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/A09-attempt-00/asset.wav",
    "sha256": "fa96dafcf7bf7950302b74fe4997ef448448da2be34bd2ff945538534454e350",
    "bytes": 420524
  },
  "native_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/A09-attempt-00/asset.wav",
    "sha256": "fa96dafcf7bf7950302b74fe4997ef448448da2be34bd2ff945538534454e350",
    "bytes": 420524
  },
  "acceptance_receipt": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/A09-attempt-00/receipt.json",
    "sha256": "69525170da853cf10f4b3dffb6700dbb615a55bf3cf273b1d912b8f5a4a70fdb",
    "bytes": 2358
  },
  "observed_media": {
    "format": "PCM WAV",
    "channels": 1,
    "sample_rate_hz": 24000,
    "sample_width_bytes": 2,
    "duration_seconds": 8.76
  },
  "generator_id": "chatterbox_turbo",
  "prompt_field": "input",
  "seed": null,
  "seed_status": "Not exposed by the installed speech API; no seed or deterministic regeneration claim.",
  "submitted_settings": {
    "language": "en"
  },
  "fixture_or_configured_settings": {
    "language": "en",
    "voice": "bundled default",
    "voice_cloning": false,
    "format": "PCM WAV",
    "max_text_chars": 500,
    "max_duration_seconds": 30,
    "seed": null,
    "seed_availability": "Not exposed through installed speech API; no reproducibility claim."
  },
  "generation_request": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/A09-attempt-00/generation.request.json",
    "sha256": "c2906ff23971feed0497761e745388dde18d3d9d6f50c274f7866facb587632e",
    "bytes": 230
  },
  "recorded_runtime_identity_source": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/A09-attempt-00/generation.json",
    "sha256": "3f3859c14454928fd187e7f8a60fbe81c3fe76239efec2cc511a6c4f8a7fb20f",
    "bytes": 5399
  },
  "request_vs_configuration_notice": "Submitted settings and recorded runtime/fixture settings are separate evidence; a prepared generation record alone is not proof of execution. The selected asset and pre-inference acceptance receipt establish the retained input."
}
  

A10:満足している商品の使い方

A10:元のWAV全長、7.800秒。 元のファイル

想定した答え:Ordinary inquiry

商品に満足していると伝え、使い方を尋ねています。

質問と選択肢

モデルに伝えた説明

  Assess only the audible customer message. A complaint expresses dissatisfaction with a prior service or product outcome, an unmet obligation or error, or seeks a remedy for a failure. Ordinary inquiries ask factual, prospective, or status questions without asserting failure. Politeness does not remove a complaint.
  

質問

  Is the audible customer message a complaint or an ordinary inquiry?
  

元の順番

  ["Complaint", "Ordinary inquiry"]
  

逆順

  ["Ordinary inquiry", "Complaint"]
  
モデル元の順番の回答逆順の回答想定回答との一致:元 / 逆
Jev-OmniOrdinary inquiryOrdinary inquiry一致 / 一致
選択肢Jev-Omni:元Jev-Omni:逆
Complaint0.0003300.038683
Ordinary inquiry0.9996700.961316
保存済み回答と実験ID
生成プロンプトと設定
  I am happy with the speaker and I am not complaining. I just need a little help pairing it with my tablet. Which button should I hold to start pairing?
  
  {
  "fixture_id": "A10",
  "cohorts": [
    "E1_JEV_INITIAL"
  ],
  "modality": "audio",
  "selected_attempt_id": "A10-attempt-00",
  "selected_attempt_index": 0,
  "evaluated_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/A10-attempt-00/asset.wav",
    "sha256": "442eddd9ccc75ba77235d0feb6cef7259201b61f05a542cdfa87fc2bc6e2633e",
    "bytes": 374444
  },
  "native_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/A10-attempt-00/asset.wav",
    "sha256": "442eddd9ccc75ba77235d0feb6cef7259201b61f05a542cdfa87fc2bc6e2633e",
    "bytes": 374444
  },
  "acceptance_receipt": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/A10-attempt-00/receipt.json",
    "sha256": "527db193157f6a7cbe2c43112f1b771c75187410b012eb41ece1915064e89ae5",
    "bytes": 1569
  },
  "observed_media": {
    "format": "PCM WAV",
    "channels": 1,
    "sample_rate_hz": 24000,
    "sample_width_bytes": 2,
    "duration_seconds": 7.8
  },
  "generator_id": "chatterbox_turbo",
  "prompt_field": "input",
  "seed": null,
  "seed_status": "Not exposed by the installed speech API; no seed or deterministic regeneration claim.",
  "submitted_settings": {
    "language": "en"
  },
  "fixture_or_configured_settings": {
    "language": "en",
    "voice": "bundled default",
    "voice_cloning": false,
    "format": "PCM WAV",
    "max_text_chars": 500,
    "max_duration_seconds": 30,
    "seed": null,
    "seed_availability": "Not exposed through installed speech API; no reproducibility claim."
  },
  "generation_request": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/A10-attempt-00/generation.request.json",
    "sha256": "0b27bafa5a2b73cdd2a6628300ad10828a0b6b7ff05fd1407bad1ac20f41daef",
    "bytes": 215
  },
  "recorded_runtime_identity_source": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/A10-attempt-00/generation.json",
    "sha256": "7fc58929b760c5c89f0a4942ac17500a3947c5a15fec5704c2a6159a80526419",
    "bytes": 5384
  },
  "request_vs_configuration_notice": "Submitted settings and recorded runtime/fixture settings are separate evidence; a prepared generation record alone is not proof of execution. The selected asset and pre-inference acceptance receipt establish the retained input."
}
  

画像10件:写っている物を選ぶ

画像の主な物体を、4つの選択肢から選ばせました。下の画像は、モデルに渡したJPEGです。

I06のザルでは、Jev-Omniの答えがColander(ザル)からSaucepan(鍋)に変わりました。Clefはどちらの順番でもColanderを選びました。

I01:リンゴ

I01: リンゴ
I01:評価に使用したファイル。 元のファイル

想定した答え:Apple

果実の形、木質の軸、つやのある皮がリンゴの特徴に合っています。

質問と選択肢

モデルに伝えた説明

  Identify the single principal foreground object using its visible shape and diagnostic features.
  

質問

  Which option identifies the principal visible object?
  

元の順番

  ["Apple", "Pear", "Tomato", "Peach"]
  

逆順

  ["Peach", "Tomato", "Pear", "Apple"]
  
モデル元の順番の回答逆順の回答想定回答との一致:元 / 逆
Jev-OmniAppleApple一致 / 一致
ClefAppleApple一致 / 一致
選択肢Jev-Omni:元Jev-Omni:逆Clef:元Clef:逆
Apple0.9997270.9997590.9860450.988375
Pear0.0000780.0001900.0055080.003795
Tomato0.0000220.0000280.0042900.003915
Peach0.0001730.0000240.0041580.003915
保存済み回答と実験ID
生成プロンプトと設定
  A single red-green apple resting on a pale wooden kitchen counter near a window. Clearly show the apple rounded fruit body, shallow top dimple and short brown woody stem, smooth waxy skin; keep the whole fruit visible. Soft daylight, close product photograph. Only this object is salient. No text, letters, words, captions, logos, labels, watermarks, comparison panels or other candidate objects.
  
  {
  "fixture_id": "I01",
  "cohorts": [
    "E1_JEV_INITIAL",
    "E2_CLEF_COMMON"
  ],
  "modality": "image",
  "selected_attempt_id": "I01-attempt-00",
  "selected_attempt_index": 0,
  "evaluated_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/I01-attempt-00/asset.jpg",
    "sha256": "4315bc149771bfd736a920262f0880ca714d6d6ac968e7f26c27e640b978638d",
    "bytes": 87892
  },
  "native_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/I01-attempt-00/native.png",
    "sha256": "61903b8cbea29daedff7915dd2270048a4b4a18d829ea3d6d579fd7283836c3d",
    "bytes": 917357
  },
  "acceptance_receipt": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/I01-attempt-00/receipt.json",
    "sha256": "74c71f861337dc3579797b017c8e4b8cb9ab83ea7bc0b11437931d192206cb82",
    "bytes": 1296
  },
  "observed_media": {
    "native_width": 1024,
    "native_height": 1024,
    "dimension_source": "independent receipt when recorded; otherwise configured dimensions",
    "evaluated_format": "JPEG",
    "jpeg_quality": 90,
    "native_format": "PNG"
  },
  "generator_id": "sdxl_lightning_4step",
  "prompt_field": "prompt",
  "seed": 2026100201,
  "seed_status": "requested; no cross-hardware determinism claimed",
  "submitted_settings": {
    "size": "1024x1024",
    "seed": 2026100201,
    "num_inference_steps": 4,
    "guidance_scale": 0,
    "response_format": "url"
  },
  "fixture_or_configured_settings": {
    "format": "PNG",
    "seed": 2026100201,
    "steps": 4,
    "width": 1024,
    "height": 1024,
    "guidance_scale": 0,
    "scheduler": "EulerDiscreteScheduler",
    "timestep_spacing": "trailing"
  },
  "generation_request": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/I01-attempt-00/generation.request.json",
    "sha256": "387292ebf2b24f3990c0aee84699ead4128f2d3018d877c14004a4e355fde2de",
    "bytes": 564
  },
  "recorded_runtime_identity_source": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/I01-attempt-00/generation.json",
    "sha256": "bae00b6b89b26600141479b43bcabf35cf6c63126f0fe0ccde616887593c0618",
    "bytes": 31259
  },
  "request_vs_configuration_notice": "Submitted settings and recorded runtime/fixture settings are separate evidence; a prepared generation record alone is not proof of execution. The selected asset and pre-inference acceptance receipt establish the retained input."
}
  

I02:陶器のティーポット

I02: 陶器のティーポット
I02:評価に使用したファイル。 元のファイル

想定した答え:Ceramic teapot

陶器の本体に、ふた、注ぎ口、持ち手があります。

質問と選択肢

モデルに伝えた説明

  Identify the single principal foreground object using its visible shape and diagnostic features.
  

質問

  Which option identifies the principal visible object?
  

元の順番

  ["Electric kettle", "Ceramic teapot", "Watering can", "Moka pot"]
  

逆順

  ["Moka pot", "Watering can", "Ceramic teapot", "Electric kettle"]
  
モデル元の順番の回答逆順の回答想定回答との一致:元 / 逆
Jev-OmniCeramic teapotCeramic teapot一致 / 一致
ClefCeramic teapotCeramic teapot一致 / 一致
選択肢Jev-Omni:元Jev-Omni:逆Clef:元Clef:逆
Electric kettle0.0001350.0014260.0099080.004438
Ceramic teapot0.9989720.9959850.9719160.980958
Watering can0.0002000.0024100.0146430.010983
Moka pot0.0006930.0001790.0035330.003622
保存済み回答と実験ID
生成プロンプトと設定
  A single ivory glazed ceramic teapot on a dark dining table, three-quarter view. Clearly show its round ceramic body, removable lid with knob, curved pouring spout and loop handle, with no electrical cord or base. Natural afternoon light, realistic still life photograph. Only this object is salient. No text, letters, words, captions, logos, labels, watermarks, comparison panels or other candidate objects.
  
  {
  "fixture_id": "I02",
  "cohorts": [
    "E1_JEV_INITIAL",
    "E2_CLEF_COMMON"
  ],
  "modality": "image",
  "selected_attempt_id": "I02-attempt-00",
  "selected_attempt_index": 0,
  "evaluated_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/I02-attempt-00/asset.jpg",
    "sha256": "fd04ead97f99532645ff4205bb5ef7a4a60f718bc924956165404a2e74447f72",
    "bytes": 72057
  },
  "native_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/I02-attempt-00/native.png",
    "sha256": "902e7bb898901204b5e9cb9352e3ebf5bc46cdc9f9927ebc70a1854faa4419c8",
    "bytes": 893459
  },
  "acceptance_receipt": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/I02-attempt-00/receipt.json",
    "sha256": "92ae18ae18f11633424ee64e0283a87bd7fbb7a50d520805ff7db7f9b34dc7ef",
    "bytes": 1279
  },
  "observed_media": {
    "native_width": 1024,
    "native_height": 1024,
    "dimension_source": "independent receipt when recorded; otherwise configured dimensions",
    "evaluated_format": "JPEG",
    "jpeg_quality": 90,
    "native_format": "PNG"
  },
  "generator_id": "sdxl_lightning_4step",
  "prompt_field": "prompt",
  "seed": 2026100202,
  "seed_status": "requested; no cross-hardware determinism claimed",
  "submitted_settings": {
    "size": "1024x1024",
    "seed": 2026100202,
    "num_inference_steps": 4,
    "guidance_scale": 0,
    "response_format": "url"
  },
  "fixture_or_configured_settings": {
    "format": "PNG",
    "seed": 2026100202,
    "steps": 4,
    "width": 1024,
    "height": 1024,
    "guidance_scale": 0,
    "scheduler": "EulerDiscreteScheduler",
    "timestep_spacing": "trailing"
  },
  "generation_request": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/I02-attempt-00/generation.request.json",
    "sha256": "f284a29e960718cd69c04732151f510046cb538b347e7b6579943d0a882fe212",
    "bytes": 576
  },
  "recorded_runtime_identity_source": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/I02-attempt-00/generation.json",
    "sha256": "d8a3ff8a9ee5bf6db72e524bd2d28a40fce5e4317fd5e90c94552a2809e058c1",
    "bytes": 31271
  },
  "request_vs_configuration_notice": "Submitted settings and recorded runtime/fixture settings are separate evidence; a prepared generation record alone is not proof of execution. The selected asset and pre-inference acceptance receipt establish the retained input."
}
  

I03:釘抜き付きハンマー

I03: 釘抜き付きハンマー
I03:評価に使用したファイル。 元のファイル

想定した答え:Claw hammer

平らな打撃面の反対側に、二股の釘抜きがあります。

質問と選択肢

モデルに伝えた説明

  Identify the single principal foreground object using its visible shape and diagnostic features.
  

質問

  Which option identifies the principal visible object?
  

元の順番

  ["Mallet", "Pickaxe", "Claw hammer", "Screwdriver"]
  

逆順

  ["Screwdriver", "Claw hammer", "Pickaxe", "Mallet"]
  
モデル元の順番の回答逆順の回答想定回答との一致:元 / 逆
Jev-OmniClaw hammerClaw hammer一致 / 一致
ClefClaw hammerClaw hammer一致 / 一致
選択肢Jev-Omni:元Jev-Omni:逆Clef:元Clef:逆
Mallet0.0233930.0021800.0051120.008994
Pickaxe0.0562270.0006860.0026940.003991
Claw hammer0.8847090.9965380.9895830.984186
Screwdriver0.0356700.0005960.0026110.002830
保存済み回答と実験ID
生成プロンプトと設定
  A single steel claw hammer with a wooden handle lying diagonally on a clean workshop bench. Clearly show one flat striking face and an opposite curved split claw, with the full handle visible. Detailed realistic photograph, side view. Only this object is salient. No text, letters, words, captions, logos, labels, watermarks, comparison panels or other candidate objects.
  
  {
  "fixture_id": "I03",
  "cohorts": [
    "E1_JEV_INITIAL",
    "E2_CLEF_COMMON"
  ],
  "modality": "image",
  "selected_attempt_id": "I03-attempt-02",
  "selected_attempt_index": 2,
  "evaluated_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/I03-attempt-02/asset.jpg",
    "sha256": "55f1e36f8b2b10df9e640424b3ac47b01d3609c709bc8f72c6a3fcbd57ab02a1",
    "bytes": 175434
  },
  "native_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/I03-attempt-02/native.png",
    "sha256": "ebbdd19af1be1b72f43124e8fbaa81aab0d6449fc05843fa5661a210be79e394",
    "bytes": 1398723
  },
  "acceptance_receipt": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/I03-attempt-02/receipt.json",
    "sha256": "5038bb00814fc84d81f87addaa16961b53f6310d06c82664d0c50ac6fd32b4e3",
    "bytes": 1296
  },
  "observed_media": {
    "native_width": 1024,
    "native_height": 1024,
    "dimension_source": "independent receipt when recorded; otherwise configured dimensions",
    "evaluated_format": "JPEG",
    "jpeg_quality": 90,
    "native_format": "PNG"
  },
  "generator_id": "sdxl_lightning_4step",
  "prompt_field": "prompt",
  "seed": 2026102203,
  "seed_status": "requested; no cross-hardware determinism claimed",
  "submitted_settings": {
    "size": "1024x1024",
    "seed": 2026102203,
    "num_inference_steps": 4,
    "guidance_scale": 0,
    "response_format": "url"
  },
  "fixture_or_configured_settings": {
    "format": "PNG",
    "seed": 2026102203,
    "steps": 4,
    "width": 1024,
    "height": 1024,
    "guidance_scale": 0,
    "scheduler": "EulerDiscreteScheduler",
    "timestep_spacing": "trailing"
  },
  "generation_request": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/I03-attempt-02/generation.request.json",
    "sha256": "b4f8a36caea0535b95dd864466ecd567f0d305410cd87b5f4b0bc29a7ab9784f",
    "bytes": 539
  },
  "recorded_runtime_identity_source": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/I03-attempt-02/generation.json",
    "sha256": "72b7323cbe39bc6aef373154aa5bf609ce0ee5d90910da8b0091dafa2a3ff96b",
    "bytes": 31234
  },
  "request_vs_configuration_notice": "Submitted settings and recorded runtime/fixture settings are separate evidence; a prepared generation record alone is not proof of execution. The selected asset and pre-inference acceptance receipt establish the retained input."
}
  

I04:自転車

I04: 自転車
I04:評価に使用したファイル。 元のファイル

想定した答え:Bicycle

2つの車輪、フレーム、ペダル、チェーンがあり、モーターはありません。

質問と選択肢

モデルに伝えた説明

  Identify the single principal foreground object using its visible shape and diagnostic features.
  

質問

  Which option identifies the principal visible object?
  

元の順番

  ["Motorcycle", "Kick scooter", "Tricycle", "Bicycle"]
  

逆順

  ["Bicycle", "Tricycle", "Kick scooter", "Motorcycle"]
  
モデル元の順番の回答逆順の回答想定回答との一致:元 / 逆
Jev-OmniBicycleBicycle一致 / 一致
ClefBicycleBicycle一致 / 一致
選択肢Jev-Omni:元Jev-Omni:逆Clef:元Clef:逆
Motorcycle0.0000130.0001670.0038240.004034
Kick scooter0.0000280.0000220.0033740.003560
Tricycle0.0001600.0000730.0046120.005344
Bicycle0.9997990.9997380.9881900.987062
保存済み回答と実験ID
生成プロンプトと設定
  A single ordinary pedal bicycle parked beside a brick courtyard wall, entire bicycle visible from the side. Clearly show exactly two wheels, triangular frame, pedals and chain, saddle and handlebars, no motor. Overcast daylight, realistic photograph. Only this object is salient. No text, letters, words, captions, logos, labels, watermarks, comparison panels or other candidate objects.
  
  {
  "fixture_id": "I04",
  "cohorts": [
    "E1_JEV_INITIAL",
    "E2_CLEF_COMMON"
  ],
  "modality": "image",
  "selected_attempt_id": "I04-attempt-00",
  "selected_attempt_index": 0,
  "evaluated_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/I04-attempt-00/asset.jpg",
    "sha256": "0f07aec0990918bb711a076b15e87a3729dc7161818d1b8782b1e75f31c18a6b",
    "bytes": 317955
  },
  "native_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/I04-attempt-00/native.png",
    "sha256": "4fc3d4e678c178e1cfbd13680a4bdf4e21d6c86a6bc6e06402088d792884958c",
    "bytes": 1805676
  },
  "acceptance_receipt": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/I04-attempt-00/receipt.json",
    "sha256": "c718e983d710a3e716d64878ba45d37245830843971c3d716769cb544a9637e3",
    "bytes": 1309
  },
  "observed_media": {
    "native_width": 1024,
    "native_height": 1024,
    "dimension_source": "independent receipt when recorded; otherwise configured dimensions",
    "evaluated_format": "JPEG",
    "jpeg_quality": 90,
    "native_format": "PNG"
  },
  "generator_id": "sdxl_lightning_4step",
  "prompt_field": "prompt",
  "seed": 2026100204,
  "seed_status": "requested; no cross-hardware determinism claimed",
  "submitted_settings": {
    "size": "1024x1024",
    "seed": 2026100204,
    "num_inference_steps": 4,
    "guidance_scale": 0,
    "response_format": "url"
  },
  "fixture_or_configured_settings": {
    "format": "PNG",
    "seed": 2026100204,
    "steps": 4,
    "width": 1024,
    "height": 1024,
    "guidance_scale": 0,
    "scheduler": "EulerDiscreteScheduler",
    "timestep_spacing": "trailing"
  },
  "generation_request": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/I04-attempt-00/generation.request.json",
    "sha256": "f8faebb0705b5d97435eb0fa3e0bb73bcb83e7450bfafa614f02b6498348e35f",
    "bytes": 555
  },
  "recorded_runtime_identity_source": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/I04-attempt-00/generation.json",
    "sha256": "fa20664a56043e029cd4390e77c7ef76f541a2362be361e15174ce55a35d674f",
    "bytes": 31250
  },
  "request_vs_configuration_notice": "Submitted settings and recorded runtime/fixture settings are separate evidence; a prepared generation record alone is not proof of execution. The selected asset and pre-inference acceptance receipt establish the retained input."
}
  

I05:双眼鏡

I05: 双眼鏡
I05:評価に使用したファイル。 元のファイル

想定した答え:Binoculars

2つの筒が中央でつながり、接眼部も2つあります。

質問と選択肢

モデルに伝えた説明

  Identify the single principal foreground object using its visible shape and diagnostic features.
  

質問

  Which option identifies the principal visible object?
  

元の順番

  ["Binoculars", "Camera", "VR goggles", "Telescope"]
  

逆順

  ["Telescope", "VR goggles", "Camera", "Binoculars"]
  
モデル元の順番の回答逆順の回答想定回答との一致:元 / 逆
Jev-OmniBinocularsBinoculars一致 / 一致
ClefBinocularsBinoculars一致 / 一致
選択肢Jev-Omni:元Jev-Omni:逆Clef:元Clef:逆
Binoculars0.9996290.9999160.9871920.991572
Camera0.0001010.0000570.0047540.002273
VR goggles0.0000170.0000190.0035880.002742
Telescope0.0002530.0000080.0044660.003412
保存済み回答と実験ID
生成プロンプトと設定
  A single black pair of binoculars on a picnic blanket, three-quarter view. Clearly show two parallel optical barrels joined by a central hinge and focus wheel, two front objective lenses and two rear eyepieces. Crisp outdoor photograph. Only this object is salient. No text, letters, words, captions, logos, labels, watermarks, comparison panels or other candidate objects.
  
  {
  "fixture_id": "I05",
  "cohorts": [
    "E1_JEV_INITIAL",
    "E2_CLEF_COMMON"
  ],
  "modality": "image",
  "selected_attempt_id": "I05-attempt-00",
  "selected_attempt_index": 0,
  "evaluated_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/I05-attempt-00/asset.jpg",
    "sha256": "e3a5ebdc541e62cb65ebddf267205b9a7f581a7b9e671c978b7c0fea2efbf0ba",
    "bytes": 153600
  },
  "native_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/I05-attempt-00/native.png",
    "sha256": "3752690855410af9813e6f9474afda490e68a0d108d9024edf1bc70e3e2158f1",
    "bytes": 1235886
  },
  "acceptance_receipt": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/I05-attempt-00/receipt.json",
    "sha256": "4d17193ec7704dd1b627f44902e1e49bce44e7336cc3b73f20475ca9c50f4f19",
    "bytes": 1368
  },
  "observed_media": {
    "native_width": 1024,
    "native_height": 1024,
    "dimension_source": "independent receipt when recorded; otherwise configured dimensions",
    "evaluated_format": "JPEG",
    "jpeg_quality": 90,
    "native_format": "PNG"
  },
  "generator_id": "sdxl_lightning_4step",
  "prompt_field": "prompt",
  "seed": 2026100205,
  "seed_status": "requested; no cross-hardware determinism claimed",
  "submitted_settings": {
    "size": "1024x1024",
    "seed": 2026100205,
    "num_inference_steps": 4,
    "guidance_scale": 0,
    "response_format": "url"
  },
  "fixture_or_configured_settings": {
    "format": "PNG",
    "seed": 2026100205,
    "steps": 4,
    "width": 1024,
    "height": 1024,
    "guidance_scale": 0,
    "scheduler": "EulerDiscreteScheduler",
    "timestep_spacing": "trailing"
  },
  "generation_request": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/I05-attempt-00/generation.request.json",
    "sha256": "c87ac9cac8415fb843244998f6e6ca8d2f4f1077f59eec58c5901a0deb8189cf",
    "bytes": 541
  },
  "recorded_runtime_identity_source": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/I05-attempt-00/generation.json",
    "sha256": "b55ffe7fa6424cd7e62c25b8d81e63bfd472396d25913f0e30e3a43b38577aab",
    "bytes": 31236
  },
  "request_vs_configuration_notice": "Submitted settings and recorded runtime/fixture settings are separate evidence; a prepared generation record alone is not proof of execution. The selected asset and pre-inference acceptance receipt establish the retained input."
}
  

I06:ザル

I06: ザル
I06:評価に使用したファイル。 元のファイル

想定した答え:Colander

穴のあるボウルに持ち手と脚があり、ザルの特徴に合っています。

質問と選択肢

モデルに伝えた説明

  Identify the single principal foreground object using its visible shape and diagnostic features.
  

質問

  Which option identifies the principal visible object?
  

元の順番

  ["Mixing bowl", "Colander", "Fine-mesh sieve", "Saucepan"]
  

逆順

  ["Saucepan", "Fine-mesh sieve", "Colander", "Mixing bowl"]
  
モデル元の順番の回答逆順の回答想定回答との一致:元 / 逆
Jev-OmniColanderSaucepan一致 / 不一致
ClefColanderColander一致 / 一致
選択肢Jev-Omni:元Jev-Omni:逆Clef:元Clef:逆
Mixing bowl0.2689250.1039560.0223170.018830
Colander0.4766100.2137200.9526540.950817
Fine-mesh sieve0.0192160.1785700.0094500.016360
Saucepan0.2352490.5037540.0155800.013993
保存済み回答と実験ID
生成プロンプトと設定
  A single stainless steel colander standing on a kitchen worktop, empty and viewed slightly from above. Clearly show its bowl shape with discrete round drainage holes across the walls and base, two side handles and a foot ring; holes are punched metal rather than mesh. Realistic photograph. Only this object is salient. No text, letters, words, captions, logos, labels, watermarks, comparison panels or other candidate objects.
  
  {
  "fixture_id": "I06",
  "cohorts": [
    "E1_JEV_INITIAL",
    "E2_CLEF_COMMON"
  ],
  "modality": "image",
  "selected_attempt_id": "I06-attempt-00",
  "selected_attempt_index": 0,
  "evaluated_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/I06-attempt-00/asset.jpg",
    "sha256": "bf43d2e6faf01aa3fba5486488fa3d58541debe9931d43647bc034f6b0d9da3f",
    "bytes": 132115
  },
  "native_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/I06-attempt-00/native.png",
    "sha256": "dd3b2aa7ecde89a437b58482bdff672515c2e8e2344d1dff5d588d4a91fb05d5",
    "bytes": 1128379
  },
  "acceptance_receipt": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/I06-attempt-00/receipt.json",
    "sha256": "cc8ed1d0054b9fccbda890e1492282a12beb8969c7ccbaae494f3851f916c38e",
    "bytes": 1338
  },
  "observed_media": {
    "native_width": 1024,
    "native_height": 1024,
    "dimension_source": "independent receipt when recorded; otherwise configured dimensions",
    "evaluated_format": "JPEG",
    "jpeg_quality": 90,
    "native_format": "PNG"
  },
  "generator_id": "sdxl_lightning_4step",
  "prompt_field": "prompt",
  "seed": 2026100206,
  "seed_status": "requested; no cross-hardware determinism claimed",
  "submitted_settings": {
    "size": "1024x1024",
    "seed": 2026100206,
    "num_inference_steps": 4,
    "guidance_scale": 0,
    "response_format": "url"
  },
  "fixture_or_configured_settings": {
    "format": "PNG",
    "seed": 2026100206,
    "steps": 4,
    "width": 1024,
    "height": 1024,
    "guidance_scale": 0,
    "scheduler": "EulerDiscreteScheduler",
    "timestep_spacing": "trailing"
  },
  "generation_request": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/I06-attempt-00/generation.request.json",
    "sha256": "f41e493e7efe1e50c3805b88b09cc285072b90218e95a5c8514c849c7ee17f27",
    "bytes": 595
  },
  "recorded_runtime_identity_source": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/I06-attempt-00/generation.json",
    "sha256": "7a08671a999d7d4b2c539884b3990f7aff418f7916eaf004afb3f4277d319727",
    "bytes": 31290
  },
  "request_vs_configuration_notice": "Submitted settings and recorded runtime/fixture settings are separate evidence; a prepared generation record alone is not proof of execution. The selected asset and pre-inference acceptance receipt establish the retained input."
}
  

I07:磁気コンパス

I07: 磁気コンパス
I07:評価に使用したファイル。 元のファイル

想定した答え:Magnetic compass

向きを示す針とケースが見えます。

質問と選択肢

モデルに伝えた説明

  Identify the single principal foreground object using its visible shape and diagnostic features.
  

質問

  Which option identifies the principal visible object?
  

元の順番

  ["Wristwatch", "Barometer", "Magnetic compass", "Thermometer"]
  

逆順

  ["Thermometer", "Magnetic compass", "Barometer", "Wristwatch"]
  
モデル元の順番の回答逆順の回答想定回答との一致:元 / 逆
Jev-OmniMagnetic compassMagnetic compass一致 / 一致
ClefMagnetic compassMagnetic compass一致 / 一致
選択肢Jev-Omni:元Jev-Omni:逆Clef:元Clef:逆
Wristwatch0.0054070.0030930.0022420.002847
Barometer0.0405010.0011650.0023860.003892
Magnetic compass0.9381390.9947770.9933300.990230
Thermometer0.0159540.0009660.0020410.003031
保存済み回答と実験ID
生成プロンプトと設定
  A single open brass pocket magnetic compass on a flat rock outdoors. Clearly show a circular directional dial with simple untitled tick marks and a freely pivoting red-and-white needle pointing across it, a hinged cover and no watch hands, wrist strap or numeric readout. Close realistic photograph. Only this object is salient. No text, letters, words, captions, logos, labels, watermarks, comparison panels or other candidate objects.
  
  {
  "fixture_id": "I07",
  "cohorts": [
    "E1_JEV_INITIAL",
    "E2_CLEF_COMMON"
  ],
  "modality": "image",
  "selected_attempt_id": "I07-attempt-00",
  "selected_attempt_index": 0,
  "evaluated_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/I07-attempt-00/asset.jpg",
    "sha256": "7bd79a2659dac9281921a8c5304770abfecafb1b46c203d9eaf944edb130032b",
    "bytes": 230111
  },
  "native_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/I07-attempt-00/native.png",
    "sha256": "cc87cd16d34a666b2fba28d8240a43a7849015a6b8d96595528b0a056c9de999",
    "bytes": 1569534
  },
  "acceptance_receipt": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/I07-attempt-00/receipt.json",
    "sha256": "e582bd37bc07321f79741b1162cc735ceda4f872873519738163a802132c5c4e",
    "bytes": 1390
  },
  "observed_media": {
    "native_width": 1024,
    "native_height": 1024,
    "dimension_source": "independent receipt when recorded; otherwise configured dimensions",
    "evaluated_format": "JPEG",
    "jpeg_quality": 90,
    "native_format": "PNG"
  },
  "generator_id": "sdxl_lightning_4step",
  "prompt_field": "prompt",
  "seed": 2026100207,
  "seed_status": "requested; no cross-hardware determinism claimed",
  "submitted_settings": {
    "size": "1024x1024",
    "seed": 2026100207,
    "num_inference_steps": 4,
    "guidance_scale": 0,
    "response_format": "url"
  },
  "fixture_or_configured_settings": {
    "format": "PNG",
    "seed": 2026100207,
    "steps": 4,
    "width": 1024,
    "height": 1024,
    "guidance_scale": 0,
    "scheduler": "EulerDiscreteScheduler",
    "timestep_spacing": "trailing"
  },
  "generation_request": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/I07-attempt-00/generation.request.json",
    "sha256": "9b1f5fb4bcd85849b296eb85b2399dda65a126f3e90fa62463cbd219fa55222e",
    "bytes": 604
  },
  "recorded_runtime_identity_source": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/I07-attempt-00/generation.json",
    "sha256": "5a960474c1be11cabad219fa0ab9c899a15b8a6408af6da2ce6c877645f8dd8c",
    "bytes": 31299
  },
  "request_vs_configuration_notice": "Submitted settings and recorded runtime/fixture settings are separate evidence; a prepared generation record alone is not proof of execution. The selected asset and pre-inference acceptance receipt establish the retained input."
}
  

I08R:卓上扇風機

I08R: 卓上扇風機
I08R:評価に使用したファイル。 元のファイル

想定した答え:Electric desk fan

円形の保護枠の中に羽根があり、台座で支えられています。

質問と選択肢

モデルに伝えた説明

  Identify the single principal foreground object using its visible shape and diagnostic features.
  

質問

  Which option identifies the principal visible object?
  

元の順番

  ["Table lamp", "Loudspeaker", "Humidifier", "Electric desk fan"]
  

逆順

  ["Electric desk fan", "Humidifier", "Loudspeaker", "Table lamp"]
  
モデル元の順番の回答逆順の回答想定回答との一致:元 / 逆
Jev-OmniElectric desk fanElectric desk fan一致 / 一致
ClefElectric desk fanElectric desk fan一致 / 一致
選択肢Jev-Omni:元Jev-Omni:逆Clef:元Clef:逆
Table lamp0.0000150.0002890.0018610.002577
Loudspeaker0.0000260.0000350.0014950.002346
Humidifier0.0001280.0001580.0020440.003108
Electric desk fan0.9998310.9995170.9946000.991969
保存済み回答と実験ID
生成プロンプトと設定
  A single electric desk fan on a plain desk. Clearly visible rotor blades inside a round protective grille, supported by a standing pedestal. Realistic photograph, full object visible. No text, labels, logos, watermarks or other salient objects.
  
  {
  "fixture_id": "I08R",
  "cohorts": [
    "E1_JEV_INITIAL",
    "E2_CLEF_COMMON"
  ],
  "modality": "image",
  "selected_attempt_id": "I08R-attempt-00",
  "selected_attempt_index": 0,
  "evaluated_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/I08R-attempt-00/asset.jpg",
    "sha256": "9c455bfcb3271d0bf877ff52e6ad886b53190ec090cbedbad3c17a9643c1f284",
    "bytes": 172973
  },
  "native_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/I08R-attempt-00/native.png",
    "sha256": "52b582b2de26dd7fdeab2751f20c287103ecbe6caa337fd8361939cfbb374b86",
    "bytes": 1144635
  },
  "acceptance_receipt": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/I08R-attempt-00/receipt.json",
    "sha256": "509d7010b76570c73a49b10c1af8ff686021281aa6ef6571913d68c0e9648520",
    "bytes": 1290
  },
  "observed_media": {
    "native_width": 1024,
    "native_height": 1024,
    "dimension_source": "independent receipt when recorded; otherwise configured dimensions",
    "evaluated_format": "JPEG",
    "jpeg_quality": 90,
    "native_format": "PNG"
  },
  "generator_id": "sdxl_lightning_4step",
  "prompt_field": "prompt",
  "seed": 2026100308,
  "seed_status": "requested; no cross-hardware determinism claimed",
  "submitted_settings": {
    "size": "1024x1024",
    "seed": 2026100308,
    "num_inference_steps": 4,
    "guidance_scale": 0,
    "response_format": "url"
  },
  "fixture_or_configured_settings": {
    "format": "PNG",
    "seed": 2026100308,
    "steps": 4,
    "width": 1024,
    "height": 1024,
    "guidance_scale": 0,
    "scheduler": "EulerDiscreteScheduler",
    "timestep_spacing": "trailing"
  },
  "generation_request": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/I08R-attempt-00/generation.request.json",
    "sha256": "28edd8216f0294a8cb1aa4727fd998cd2f4a3f6c17b5f7a3dcc2ee31cc00ee52",
    "bytes": 412
  },
  "recorded_runtime_identity_source": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/I08R-attempt-00/generation.json",
    "sha256": "e56b4e1cd6226e7ef135caed9e0312eb39c93ded3eb721f82d80aef27d3a4805",
    "bytes": 30154
  },
  "request_vs_configuration_notice": "Submitted settings and recorded runtime/fixture settings are separate evidence; a prepared generation record alone is not proof of execution. The selected asset and pre-inference acceptance receipt establish the retained input.",
  "replacement_lineage": {
    "replaces_fixture": "I08",
    "original_target": "Winged corkscrew",
    "new_target": "Electric desk fan",
    "authorized_before_predictions": true,
    "amendment": {
      "path": "2026-10-02-jev-omni-40-synthetic/amendment.json",
      "sha256": "294c7e0960d30a8615dd98f787dede79772bf4bf3b992b9917ad6473ee81cf42",
      "bytes": 2274
    },
    "amendment_freeze": {
      "path": "2026-10-02-jev-omni-40-synthetic/amendment.freeze.json",
      "sha256": "4f52bdf1554f2aa9c06f64754f47c84b353cdb1747cacd537102223ae6e4d005",
      "bytes": 225
    },
    "reason": "All three frozen winged-corkscrew attempts independently rejected for missing diagnostic helix/collar. User authorized one fixed everyday-object substitution before any classifier output; original protocol and rejected assets remain immutable."
  }
}
  

I09:バドミントンのシャトル

I09: バドミントンのシャトル
I09:評価に使用したファイル。 元のファイル

想定した答え:Badminton shuttlecock

丸いコルクの底の周りに羽根が並んでいます。

質問と選択肢

モデルに伝えた説明

  Identify the single principal foreground object using its visible shape and diagnostic features.
  

質問

  Which option identifies the principal visible object?
  

元の順番

  ["Badminton shuttlecock", "Feather duster", "Table-tennis ball", "Spinning top"]
  

逆順

  ["Spinning top", "Table-tennis ball", "Feather duster", "Badminton shuttlecock"]
  
モデル元の順番の回答逆順の回答想定回答との一致:元 / 逆
Jev-OmniBadminton shuttlecockBadminton shuttlecock一致 / 一致
ClefBadminton shuttlecockBadminton shuttlecock一致 / 一致
選択肢Jev-Omni:元Jev-Omni:逆Clef:元Clef:逆
Badminton shuttlecock0.9995830.9998150.9885560.990326
Feather duster0.0001230.0001290.0054360.004143
Table-tennis ball0.0000240.0000400.0030980.002938
Spinning top0.0002690.0000160.0029100.002593
保存済み回答と実験ID
生成プロンプトと設定
  A single white feather badminton shuttlecock on a blue sports-court floor, close side view. Clearly show a small rounded cork base and an open tapered crown of individually visible feathers secured in a circular skirt. Entire object in focus, realistic photograph. Only this object is salient. No text, letters, words, captions, logos, labels, watermarks, comparison panels or other candidate objects.
  
  {
  "fixture_id": "I09",
  "cohorts": [
    "E1_JEV_INITIAL",
    "E2_CLEF_COMMON"
  ],
  "modality": "image",
  "selected_attempt_id": "I09-attempt-00",
  "selected_attempt_index": 0,
  "evaluated_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/I09-attempt-00/asset.jpg",
    "sha256": "7fcf2f7a9ef0967c693a2dbcebcad84f6b58700057f4f1dd3d3c0cb1752168d0",
    "bytes": 113819
  },
  "native_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/I09-attempt-00/native.png",
    "sha256": "8b2c8c9b0f017e862b387d6f28ea3f563ea1184e1fef170f0648319d5700453e",
    "bytes": 1080435
  },
  "acceptance_receipt": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/I09-attempt-00/receipt.json",
    "sha256": "fd6d6b2ba11ebd6afc5e9f8f3577cf16bac6544b2cc15f6a17f879472b86c873",
    "bytes": 1363
  },
  "observed_media": {
    "native_width": 1024,
    "native_height": 1024,
    "dimension_source": "independent receipt when recorded; otherwise configured dimensions",
    "evaluated_format": "JPEG",
    "jpeg_quality": 90,
    "native_format": "PNG"
  },
  "generator_id": "sdxl_lightning_4step",
  "prompt_field": "prompt",
  "seed": 2026100209,
  "seed_status": "requested; no cross-hardware determinism claimed",
  "submitted_settings": {
    "size": "1024x1024",
    "seed": 2026100209,
    "num_inference_steps": 4,
    "guidance_scale": 0,
    "response_format": "url"
  },
  "fixture_or_configured_settings": {
    "format": "PNG",
    "seed": 2026100209,
    "steps": 4,
    "width": 1024,
    "height": 1024,
    "guidance_scale": 0,
    "scheduler": "EulerDiscreteScheduler",
    "timestep_spacing": "trailing"
  },
  "generation_request": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/I09-attempt-00/generation.request.json",
    "sha256": "ba913aeea879195372a8e23a66207edd0f24c7fb70929ca21f396ff7d2df6c2a",
    "bytes": 569
  },
  "recorded_runtime_identity_source": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/I09-attempt-00/generation.json",
    "sha256": "e5493587362c21658dc6cf800c492e2127701ccd5a973da29a620badbaa87550",
    "bytes": 31264
  },
  "request_vs_configuration_notice": "Submitted settings and recorded runtime/fixture settings are separate evidence; a prepared generation record alone is not proof of execution. The selected asset and pre-inference acceptance receipt establish the retained input."
}
  

I10:テレビのリモコン

I10: テレビのリモコン
I10:評価に使用したファイル。 元のファイル

想定した答え:Television remote control

細長い本体にボタン、方向キー、赤外線の先端部があります。

質問と選択肢

モデルに伝えた説明

  Identify the single principal foreground object using its visible shape and diagnostic features.
  

質問

  Which option identifies the principal visible object?
  

元の順番

  ["Calculator", "Television remote control", "Mobile phone", "Game controller"]
  

逆順

  ["Game controller", "Mobile phone", "Television remote control", "Calculator"]
  
モデル元の順番の回答逆順の回答想定回答との一致:元 / 逆
Jev-OmniTelevision remote controlTelevision remote control一致 / 一致
ClefTelevision remote controlTelevision remote control一致 / 一致
選択肢Jev-Omni:元Jev-Omni:逆Clef:元Clef:逆
Calculator0.0001310.0011220.0025890.003919
Television remote control0.9991120.9961120.9887250.989335
Mobile phone0.0002690.0025370.0035380.002164
Game controller0.0004880.0002300.0051480.004582
保存済み回答と実験ID
生成プロンプトと設定
  A single matte black television remote control on a grey sofa cushion, full length visible from above. Clearly show an elongated narrow handheld body, many rubber buttons including a circular directional pad, colored buttons and a small infrared emitter at the top, with no screen or joysticks. Realistic photograph. Only this object is salient. No text, letters, words, captions, logos, labels, watermarks, comparison panels or other candidate objects.
  
  {
  "fixture_id": "I10",
  "cohorts": [
    "E1_JEV_INITIAL",
    "E2_CLEF_COMMON"
  ],
  "modality": "image",
  "selected_attempt_id": "I10-attempt-00",
  "selected_attempt_index": 0,
  "evaluated_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/I10-attempt-00/asset.jpg",
    "sha256": "1c1d8a0e11414eb2ecb0cded580f1200ed7a14bbe70fd1d845220e2723e290e4",
    "bytes": 222750
  },
  "native_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/I10-attempt-00/native.png",
    "sha256": "d4635642744dfd4845bb4b96dfc93a43219610025c30e401afd9b7120e27fd02",
    "bytes": 1610855
  },
  "acceptance_receipt": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/I10-attempt-00/receipt.json",
    "sha256": "72bd07dc13b97009ed162c09b68ee69a14f4e9eb90bb42b7454b2c208d86c441",
    "bytes": 1359
  },
  "observed_media": {
    "native_width": 1024,
    "native_height": 1024,
    "dimension_source": "independent receipt when recorded; otherwise configured dimensions",
    "evaluated_format": "JPEG",
    "jpeg_quality": 90,
    "native_format": "PNG"
  },
  "generator_id": "sdxl_lightning_4step",
  "prompt_field": "prompt",
  "seed": 2026100210,
  "seed_status": "requested; no cross-hardware determinism claimed",
  "submitted_settings": {
    "size": "1024x1024",
    "seed": 2026100210,
    "num_inference_steps": 4,
    "guidance_scale": 0,
    "response_format": "url"
  },
  "fixture_or_configured_settings": {
    "format": "PNG",
    "seed": 2026100210,
    "steps": 4,
    "width": 1024,
    "height": 1024,
    "guidance_scale": 0,
    "scheduler": "EulerDiscreteScheduler",
    "timestep_spacing": "trailing"
  },
  "generation_request": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/I10-attempt-00/generation.request.json",
    "sha256": "2fd12d6c46c75a127d85c881b212d8aa941f6287ee70a9c5b6f458b202e7dc57",
    "bytes": 621
  },
  "recorded_runtime_identity_source": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/I10-attempt-00/generation.json",
    "sha256": "027e8ceeb27d464db91eb317ce194d493cf4a6a02014de4493f436306e60a5b2",
    "bytes": 31316
  },
  "request_vs_configuration_notice": "Submitted settings and recorded runtime/fixture settings are separate evidence; a prepared generation record alone is not proof of execution. The selected asset and pre-inference acceptance receipt establish the retained input."
}
  

日本語メール10件:返信が必要か

実験用に書いた架空の日本語メールを使いました。質問や対応の依頼が残っていればReply required、受領の連絡やお礼、解決済みの依頼だけならNo reply requiredとしました。

T01では、選択肢を逆の順番にすると、Jev-Omniの答えが「返信が必要」から「返信は不要」に変わりました。Clefはどちらでも「返信が必要」を選びました。残る9件は、両モデルとも両方の順番で想定した答えと一致しました。

T01:展示会の受付時間と予約

  件名:展示会の受付時間について

架空ストア ご担当者様

来週の展示会への参加を検討しています。土曜日の受付は何時から何時まででしょうか。また、予約なしでも入場できますか。

お手数ですが、この二点について教えてください。

水野
  

入力ファイル

想定した答え:Reply required

受付時間と入場方法について質問しています。

質問と選択肢

モデルに伝えた説明

  Classify only the following fictional customer email. A reply is required for an outstanding explicit request for an answer, remedy, decision, approval, confirmation, or specified follow-up. Acknowledgments and information with no outstanding request do not require a reply. A quoted old request alone does not create a new obligation.

Email:
件名:展示会の受付時間について

架空ストア ご担当者様

来週の展示会への参加を検討しています。土曜日の受付は何時から何時まででしょうか。また、予約なしでも入場できますか。

お手数ですが、この二点について教えてください。

水野
  

質問

  Does this email require a human reply?
  

元の順番

  ["Reply required", "No reply required"]
  

逆順

  ["No reply required", "Reply required"]
  
モデル元の順番の回答逆順の回答想定回答との一致:元 / 逆
Jev-OmniReply requiredNo reply required一致 / 不一致
ClefReply requiredReply required一致 / 一致
選択肢Jev-Omni:元Jev-Omni:逆Clef:元Clef:逆
Reply required0.9585380.0860160.9910150.991875
No reply required0.0414620.9139840.0089850.008125
保存済み回答と実験ID
生成プロンプトと設定
  {
  "fixture_id": "T01",
  "cohorts": [
    "E1_JEV_INITIAL",
    "E2_CLEF_COMMON"
  ],
  "modality": "text",
  "selected_attempt_id": "T01-attempt-00",
  "selected_attempt_index": 0,
  "evaluated_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/T01-attempt-00/asset.txt",
    "sha256": "ad63ba8c1b406ac1cd643ed0a86b5bffa40f611b21fa927ea5c407d84f455470",
    "bytes": 330
  },
  "native_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/T01-attempt-00/asset.txt",
    "sha256": "ad63ba8c1b406ac1cd643ed0a86b5bffa40f611b21fa927ea5c407d84f455470",
    "bytes": 330
  },
  "acceptance_receipt": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/T01-attempt-00/receipt.json",
    "sha256": "c4827fa675d2508e80031ee587a4c38318041ed4fffa343e8e5c5ee2a6915f92",
    "bytes": 506
  },
  "generator_id": null,
  "generation_method": "Original fictional Japanese email authored before predictions; no generative model used.",
  "seed": null,
  "settings": null
}
  

T02:届いた商品への対応依頼

  件名:届いた商品についてのお願い

架空ストア ご担当者様

本日届いた注文番号 F-204 のマグカップにひびがありました。交換を希望します。返送方法と交換品の発送予定をお知らせください。

写真は手元にありますので、必要でしたら送ります。よろしくお願いいたします。

藤井
  

入力ファイル

想定した答え:Reply required

返品方法と交換時期を尋ねています。

質問と選択肢

モデルに伝えた説明

  Classify only the following fictional customer email. A reply is required for an outstanding explicit request for an answer, remedy, decision, approval, confirmation, or specified follow-up. Acknowledgments and information with no outstanding request do not require a reply. A quoted old request alone does not create a new obligation.

Email:
件名:届いた商品についてのお願い

架空ストア ご担当者様

本日届いた注文番号 F-204 のマグカップにひびがありました。交換を希望します。返送方法と交換品の発送予定をお知らせください。

写真は手元にありますので、必要でしたら送ります。よろしくお願いいたします。

藤井
  

質問

  Does this email require a human reply?
  

元の順番

  ["Reply required", "No reply required"]
  

逆順

  ["No reply required", "Reply required"]
  
モデル元の順番の回答逆順の回答想定回答との一致:元 / 逆
Jev-OmniReply requiredReply required一致 / 一致
ClefReply requiredReply required一致 / 一致
選択肢Jev-Omni:元Jev-Omni:逆Clef:元Clef:逆
Reply required0.9991570.9998310.9896720.990514
No reply required0.0008430.0001690.0103280.009486
保存済み回答と実験ID
生成プロンプトと設定
  {
  "fixture_id": "T02",
  "cohorts": [
    "E1_JEV_INITIAL",
    "E2_CLEF_COMMON"
  ],
  "modality": "text",
  "selected_attempt_id": "T02-attempt-00",
  "selected_attempt_index": 0,
  "evaluated_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/T02-attempt-00/asset.txt",
    "sha256": "2f059c84917dcc914502603dbd2011b2b554e6420ffb22c51ac79eb2e532dbd2",
    "bytes": 382
  },
  "native_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/T02-attempt-00/asset.txt",
    "sha256": "2f059c84917dcc914502603dbd2011b2b554e6420ffb22c51ac79eb2e532dbd2",
    "bytes": 382
  },
  "acceptance_receipt": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/T02-attempt-00/receipt.json",
    "sha256": "2c903d2a95fd4f5d20338f417232b6f68de96e46d31499d42bce2448001894c0",
    "bytes": 533
  },
  "generator_id": null,
  "generation_method": "Original fictional Japanese email authored before predictions; no generative model used.",
  "seed": null,
  "settings": null
}
  

T03:見積書の送付方法の確認

  件名:見積書の送付方法の確認

架空サービス ご担当者様

見積書を受け取りました。内容は社内で確認済みです。経理担当を宛先に追加してこの見積書を転送してもよろしいでしょうか。差し支えないか、ご承認をお願いします。

承認をいただくまで転送は控えます。

長谷川
  

入力ファイル

想定した答え:Reply required

受領の連絡に加えて、承認を求める依頼が残っています。

質問と選択肢

モデルに伝えた説明

  Classify only the following fictional customer email. A reply is required for an outstanding explicit request for an answer, remedy, decision, approval, confirmation, or specified follow-up. Acknowledgments and information with no outstanding request do not require a reply. A quoted old request alone does not create a new obligation.

Email:
件名:見積書の送付方法の確認

架空サービス ご担当者様

見積書を受け取りました。内容は社内で確認済みです。経理担当を宛先に追加してこの見積書を転送してもよろしいでしょうか。差し支えないか、ご承認をお願いします。

承認をいただくまで転送は控えます。

長谷川
  

質問

  Does this email require a human reply?
  

元の順番

  ["Reply required", "No reply required"]
  

逆順

  ["No reply required", "Reply required"]
  
モデル元の順番の回答逆順の回答想定回答との一致:元 / 逆
Jev-OmniReply requiredReply required一致 / 一致
ClefReply requiredReply required一致 / 一致
選択肢Jev-Omni:元Jev-Omni:逆Clef:元Clef:逆
Reply required0.9975660.9996200.9904410.991084
No reply required0.0024340.0003800.0095590.008916
保存済み回答と実験ID
生成プロンプトと設定
  {
  "fixture_id": "T03",
  "cohorts": [
    "E1_JEV_INITIAL",
    "E2_CLEF_COMMON"
  ],
  "modality": "text",
  "selected_attempt_id": "T03-attempt-00",
  "selected_attempt_index": 0,
  "evaluated_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/T03-attempt-00/asset.txt",
    "sha256": "63a2143131c028debd55fb558b52899e14f9db26fb6af773149016f819354446",
    "bytes": 375
  },
  "native_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/T03-attempt-00/asset.txt",
    "sha256": "63a2143131c028debd55fb558b52899e14f9db26fb6af773149016f819354446",
    "bytes": 375
  },
  "acceptance_receipt": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/T03-attempt-00/receipt.json",
    "sha256": "46109acbe5f7a23b4d358dad9fd8d53d71aba62ec306ed9c8f4112991590a79e",
    "bytes": 532
  },
  "generator_id": null,
  "generation_method": "Original fictional Japanese email authored before predictions; no generative model used.",
  "seed": null,
  "settings": null
}
  

T04:資料の到着と確認の依頼

  件名:資料の到着と確認のお願い

架空サービス ご担当者様

資料を受け取りました。ありがとうございます。添付した申込内容を確認いただき、変更が必要な場合は明日正午までにご連絡ください。変更がない場合も、その旨を一言ご返信ください。

確認結果を待って手続きを進めます。

森
  

入力ファイル

想定した答え:Reply required

修正が必要でも不要でも、返信するよう求めています。

質問と選択肢

モデルに伝えた説明

  Classify only the following fictional customer email. A reply is required for an outstanding explicit request for an answer, remedy, decision, approval, confirmation, or specified follow-up. Acknowledgments and information with no outstanding request do not require a reply. A quoted old request alone does not create a new obligation.

Email:
件名:資料の到着と確認のお願い

架空サービス ご担当者様

資料を受け取りました。ありがとうございます。添付した申込内容を確認いただき、変更が必要な場合は明日正午までにご連絡ください。変更がない場合も、その旨を一言ご返信ください。

確認結果を待って手続きを進めます。

森
  

質問

  Does this email require a human reply?
  

元の順番

  ["Reply required", "No reply required"]
  

逆順

  ["No reply required", "Reply required"]
  
モデル元の順番の回答逆順の回答想定回答との一致:元 / 逆
Jev-OmniReply requiredReply required一致 / 一致
ClefReply requiredReply required一致 / 一致
選択肢Jev-Omni:元Jev-Omni:逆Clef:元Clef:逆
Reply required0.9959300.9989010.9920620.993359
No reply required0.0040700.0010990.0079380.006641
保存済み回答と実験ID
生成プロンプトと設定
  {
  "fixture_id": "T04",
  "cohorts": [
    "E1_JEV_INITIAL",
    "E2_CLEF_COMMON"
  ],
  "modality": "text",
  "selected_attempt_id": "T04-attempt-00",
  "selected_attempt_index": 0,
  "evaluated_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/T04-attempt-00/asset.txt",
    "sha256": "35636e02433c3e36c0a425444b54eb19c7a145eb933cbb59f56842303357d818",
    "bytes": 396
  },
  "native_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/T04-attempt-00/asset.txt",
    "sha256": "35636e02433c3e36c0a425444b54eb19c7a145eb933cbb59f56842303357d818",
    "bytes": 396
  },
  "acceptance_receipt": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/T04-attempt-00/receipt.json",
    "sha256": "48b620cfe9a89fbae4de5763516572060d812261a2c78ef3bcbf3b148b6107e2",
    "bytes": 530
  },
  "generator_id": null,
  "generation_method": "Original fictional Japanese email authored before predictions; no generative model used.",
  "seed": null,
  "settings": null
}
  

T05:登録情報についての新しい質問

  件名:登録情報について

架空サービス ご担当者様

先日は丁寧なご案内をありがとうございました。

> 登録時の電話番号は変更できますか。
> 会員画面から変更できます。

電話番号の変更は完了しました。なお、今回の連絡先メールアドレスの変更はまだできていません。変更手順をこのメールへの返信で教えていただけますか。

佐々木
  

入力ファイル

想定した答え:Reply required

引用した過去の問題は解決済みですが、メールアドレスについて新しい質問があります。

質問と選択肢

モデルに伝えた説明

  Classify only the following fictional customer email. A reply is required for an outstanding explicit request for an answer, remedy, decision, approval, confirmation, or specified follow-up. Acknowledgments and information with no outstanding request do not require a reply. A quoted old request alone does not create a new obligation.

Email:
件名:登録情報について

架空サービス ご担当者様

先日は丁寧なご案内をありがとうございました。

> 登録時の電話番号は変更できますか。
> 会員画面から変更できます。

電話番号の変更は完了しました。なお、今回の連絡先メールアドレスの変更はまだできていません。変更手順をこのメールへの返信で教えていただけますか。

佐々木
  

質問

  Does this email require a human reply?
  

元の順番

  ["Reply required", "No reply required"]
  

逆順

  ["No reply required", "Reply required"]
  
モデル元の順番の回答逆順の回答想定回答との一致:元 / 逆
Jev-OmniReply requiredReply required一致 / 一致
ClefReply requiredReply required一致 / 一致
選択肢Jev-Omni:元Jev-Omni:逆Clef:元Clef:逆
Reply required0.9991170.9998690.9884920.989672
No reply required0.0008830.0001310.0115080.010328
保存済み回答と実験ID
生成プロンプトと設定
  {
  "fixture_id": "T05",
  "cohorts": [
    "E1_JEV_INITIAL",
    "E2_CLEF_COMMON"
  ],
  "modality": "text",
  "selected_attempt_id": "T05-attempt-00",
  "selected_attempt_index": 0,
  "evaluated_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/T05-attempt-00/asset.txt",
    "sha256": "2e581409133ffbdbe34e5f4bc9aa95178fa79a390163d4ec9fe0d33b830ca57b",
    "bytes": 460
  },
  "native_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/T05-attempt-00/asset.txt",
    "sha256": "2e581409133ffbdbe34e5f4bc9aa95178fa79a390163d4ec9fe0d33b830ca57b",
    "bytes": 460
  },
  "acceptance_receipt": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/T05-attempt-00/receipt.json",
    "sha256": "767714b150d42dd5d1ecb3ee8913b5749f6767d1c917e73d84d5f1b372610bc8",
    "bytes": 541
  },
  "generator_id": null,
  "generation_method": "Original fictional Japanese email authored before predictions; no generative model used.",
  "seed": null,
  "settings": null
}
  

T06:資料を受け取った連絡

  件名:資料受領のご連絡

架空サービス ご担当者様

資料を確かに受け取りました。わかりやすいご説明をありがとうございます。まずは社内で共有いたします。

取り急ぎ、受領のお礼まで。

小林
  

入力ファイル

想定した答え:No reply required

受領の連絡とお礼だけで、未回答の依頼はありません。

質問と選択肢

モデルに伝えた説明

  Classify only the following fictional customer email. A reply is required for an outstanding explicit request for an answer, remedy, decision, approval, confirmation, or specified follow-up. Acknowledgments and information with no outstanding request do not require a reply. A quoted old request alone does not create a new obligation.

Email:
件名:資料受領のご連絡

架空サービス ご担当者様

資料を確かに受け取りました。わかりやすいご説明をありがとうございます。まずは社内で共有いたします。

取り急ぎ、受領のお礼まで。

小林
  

質問

  Does this email require a human reply?
  

元の順番

  ["Reply required", "No reply required"]
  

逆順

  ["No reply required", "Reply required"]
  
モデル元の順番の回答逆順の回答想定回答との一致:元 / 逆
Jev-OmniNo reply requiredNo reply required一致 / 一致
ClefNo reply requiredNo reply required一致 / 一致
選択肢Jev-Omni:元Jev-Omni:逆Clef:元Clef:逆
Reply required0.0001230.0005880.0042650.004265
No reply required0.9998770.9994120.9957350.995735
保存済み回答と実験ID
生成プロンプトと設定
  {
  "fixture_id": "T06",
  "cohorts": [
    "E1_JEV_INITIAL",
    "E2_CLEF_COMMON"
  ],
  "modality": "text",
  "selected_attempt_id": "T06-attempt-00",
  "selected_attempt_index": 0,
  "evaluated_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/T06-attempt-00/asset.txt",
    "sha256": "c3f82c557a725293b36d64c3048ec0a6fa91847c7ea08060073fe35b0a843b77",
    "bytes": 267
  },
  "native_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/T06-attempt-00/asset.txt",
    "sha256": "c3f82c557a725293b36d64c3048ec0a6fa91847c7ea08060073fe35b0a843b77",
    "bytes": 267
  },
  "acceptance_receipt": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/T06-attempt-00/receipt.json",
    "sha256": "2b16b862691f37e66412a3fe441629d3c88618817e319b026bc41656e7fcdfa9",
    "bytes": 516
  },
  "generator_id": null,
  "generation_method": "Original fictional Japanese email authored before predictions; no generative model used.",
  "seed": null,
  "settings": null
}
  

T07:交換品の受領と解決

  件名:交換品を受け取りました

架空ストア ご担当者様

交換品が届き、正常に使えることを確認しました。返送も完了し、今回の件はすべて解決しました。迅速なご対応に感謝いたします。

田中
  

入力ファイル

想定した答え:No reply required

問題への対応は終わっており、新しい依頼はありません。

質問と選択肢

モデルに伝えた説明

  Classify only the following fictional customer email. A reply is required for an outstanding explicit request for an answer, remedy, decision, approval, confirmation, or specified follow-up. Acknowledgments and information with no outstanding request do not require a reply. A quoted old request alone does not create a new obligation.

Email:
件名:交換品を受け取りました

架空ストア ご担当者様

交換品が届き、正常に使えることを確認しました。返送も完了し、今回の件はすべて解決しました。迅速なご対応に感謝いたします。

田中
  

質問

  Does this email require a human reply?
  

元の順番

  ["Reply required", "No reply required"]
  

逆順

  ["No reply required", "Reply required"]
  
モデル元の順番の回答逆順の回答想定回答との一致:元 / 逆
Jev-OmniNo reply requiredNo reply required一致 / 一致
ClefNo reply requiredNo reply required一致 / 一致
選択肢Jev-Omni:元Jev-Omni:逆Clef:元Clef:逆
Reply required0.0001630.0008560.0051400.004905
No reply required0.9998370.9991440.9948600.995095
保存済み回答と実験ID
生成プロンプトと設定
  {
  "fixture_id": "T07",
  "cohorts": [
    "E1_JEV_INITIAL",
    "E2_CLEF_COMMON"
  ],
  "modality": "text",
  "selected_attempt_id": "T07-attempt-00",
  "selected_attempt_index": 0,
  "evaluated_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/T07-attempt-00/asset.txt",
    "sha256": "c272d354f44f37bfaa99e9a4e97c94687bf8121afe9f85783c5104f47f43c8fb",
    "bytes": 265
  },
  "native_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/T07-attempt-00/asset.txt",
    "sha256": "c272d354f44f37bfaa99e9a4e97c94687bf8121afe9f85783c5104f47f43c8fb",
    "bytes": 265
  },
  "acceptance_receipt": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/T07-attempt-00/receipt.json",
    "sha256": "9b190814dea66a12494273606796632959f18167b557ed17c3ee4f8c55e64ba2",
    "bytes": 516
  },
  "generator_id": null,
  "generation_method": "Original fictional Japanese email authored before predictions; no generative model used.",
  "seed": null,
  "settings": null
}
  

T08:担当者変更のお知らせ

  件名:担当者変更のお知らせ

架空サービス ご担当者様

来月から弊社の窓口担当が吉田に変わります。連絡先一覧は本日、共有済みです。これは情報共有のみのお知らせで、返信は不要です。

引き続きよろしくお願いいたします。

中村
  

入力ファイル

想定した答え:No reply required

情報を共有し、返信は不要と伝えています。質問もありません。

質問と選択肢

モデルに伝えた説明

  Classify only the following fictional customer email. A reply is required for an outstanding explicit request for an answer, remedy, decision, approval, confirmation, or specified follow-up. Acknowledgments and information with no outstanding request do not require a reply. A quoted old request alone does not create a new obligation.

Email:
件名:担当者変更のお知らせ

架空サービス ご担当者様

来月から弊社の窓口担当が吉田に変わります。連絡先一覧は本日、共有済みです。これは情報共有のみのお知らせで、返信は不要です。

引き続きよろしくお願いいたします。

中村
  

質問

  Does this email require a human reply?
  

元の順番

  ["Reply required", "No reply required"]
  

逆順

  ["No reply required", "Reply required"]
  
モデル元の順番の回答逆順の回答想定回答との一致:元 / 逆
Jev-OmniNo reply requiredNo reply required一致 / 一致
ClefNo reply requiredNo reply required一致 / 一致
選択肢Jev-Omni:元Jev-Omni:逆Clef:元Clef:逆
Reply required0.0001560.0004730.0040390.003679
No reply required0.9998440.9995270.9959610.996322
保存済み回答と実験ID
生成プロンプトと設定
  {
  "fixture_id": "T08",
  "cohorts": [
    "E1_JEV_INITIAL",
    "E2_CLEF_COMMON"
  ],
  "modality": "text",
  "selected_attempt_id": "T08-attempt-00",
  "selected_attempt_index": 0,
  "evaluated_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/T08-attempt-00/asset.txt",
    "sha256": "6f7cfde952fe36c696c72be7d0058c25e6d1d78be5dec5c77948ab4880fc10aa",
    "bytes": 321
  },
  "native_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/T08-attempt-00/asset.txt",
    "sha256": "6f7cfde952fe36c696c72be7d0058c25e6d1d78be5dec5c77948ab4880fc10aa",
    "bytes": 321
  },
  "acceptance_receipt": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/T08-attempt-00/receipt.json",
    "sha256": "d8710369b006bc7aeaf228328884e4ca311ced837fb0d0eb60c2befdc21f728f",
    "bytes": 514
  },
  "generator_id": null,
  "generation_method": "Original fictional Japanese email authored before predictions; no generative model used.",
  "seed": null,
  "settings": null
}
  

T09:合意済みの訪問日程

  件名:訪問日程の共有

架空サービス ご担当者様

先ほどお電話で合意したとおり、訪問日時は十月十五日午後二時で確定しています。当日は受付で担当の山田をお呼びします。

このメールは確定済みの日程を記録するためのご連絡です。追加の確認事項はありません。

井上
  

入力ファイル

想定した答え:No reply required

合意済みの日程を共有しており、新しい確認は求めていません。

質問と選択肢

モデルに伝えた説明

  Classify only the following fictional customer email. A reply is required for an outstanding explicit request for an answer, remedy, decision, approval, confirmation, or specified follow-up. Acknowledgments and information with no outstanding request do not require a reply. A quoted old request alone does not create a new obligation.

Email:
件名:訪問日程の共有

架空サービス ご担当者様

先ほどお電話で合意したとおり、訪問日時は十月十五日午後二時で確定しています。当日は受付で担当の山田をお呼びします。

このメールは確定済みの日程を記録するためのご連絡です。追加の確認事項はありません。

井上
  

質問

  Does this email require a human reply?
  

元の順番

  ["Reply required", "No reply required"]
  

逆順

  ["No reply required", "Reply required"]
  
モデル元の順番の回答逆順の回答想定回答との一致:元 / 逆
Jev-OmniNo reply requiredNo reply required一致 / 一致
ClefNo reply requiredNo reply required一致 / 一致
選択肢Jev-Omni:元Jev-Omni:逆Clef:元Clef:逆
Reply required0.0002070.0010010.0058650.005911
No reply required0.9997930.9990000.9941350.994089
保存済み回答と実験ID
生成プロンプトと設定
  {
  "fixture_id": "T09",
  "cohorts": [
    "E1_JEV_INITIAL",
    "E2_CLEF_COMMON"
  ],
  "modality": "text",
  "selected_attempt_id": "T09-attempt-00",
  "selected_attempt_index": 0,
  "evaluated_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/T09-attempt-00/asset.txt",
    "sha256": "e8bfa4730c00e17275008b83872f9b4f69b614dcaba8bd8dfc84d0d0c973d1c3",
    "bytes": 372
  },
  "native_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/T09-attempt-00/asset.txt",
    "sha256": "e8bfa4730c00e17275008b83872f9b4f69b614dcaba8bd8dfc84d0d0c973d1c3",
    "bytes": 372
  },
  "acceptance_receipt": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/T09-attempt-00/receipt.json",
    "sha256": "ff4d8e0201cd7fdce5072ef78a2453fc6d61f80d3526730f8db72175e22cf52d",
    "bytes": 522
  },
  "generator_id": null,
  "generation_method": "Original fictional Japanese email authored before predictions; no generative model used.",
  "seed": null,
  "settings": null
}
  

T10:会員情報の更新完了

  件名:会員情報の更新完了

架空サービス ご担当者様

ご案内どおり操作し、会員情報の更新が完了しました。保存後の画面でも変更内容を確認できました。

> 住所の修正方法を教えていただけますか。
> 設定画面の「会員情報」から修正できます。

上記は以前のやり取りです。今回は完了のご報告のみで、新たな質問や依頼はありません。

加藤
  

入力ファイル

想定した答え:No reply required

引用した過去の依頼は解決済みで、本文に新しい依頼はありません。

質問と選択肢

モデルに伝えた説明

  Classify only the following fictional customer email. A reply is required for an outstanding explicit request for an answer, remedy, decision, approval, confirmation, or specified follow-up. Acknowledgments and information with no outstanding request do not require a reply. A quoted old request alone does not create a new obligation.

Email:
件名:会員情報の更新完了

架空サービス ご担当者様

ご案内どおり操作し、会員情報の更新が完了しました。保存後の画面でも変更内容を確認できました。

> 住所の修正方法を教えていただけますか。
> 設定画面の「会員情報」から修正できます。

上記は以前のやり取りです。今回は完了のご報告のみで、新たな質問や依頼はありません。

加藤
  

質問

  Does this email require a human reply?
  

元の順番

  ["Reply required", "No reply required"]
  

逆順

  ["No reply required", "Reply required"]
  
モデル元の順番の回答逆順の回答想定回答との一致:元 / 逆
Jev-OmniNo reply requiredNo reply required一致 / 一致
ClefNo reply requiredNo reply required一致 / 一致
選択肢Jev-Omni:元Jev-Omni:逆Clef:元Clef:逆
Reply required0.0001160.0005790.0089160.007877
No reply required0.9998840.9994210.9910840.992123
保存済み回答と実験ID
生成プロンプトと設定
  {
  "fixture_id": "T10",
  "cohorts": [
    "E1_JEV_INITIAL",
    "E2_CLEF_COMMON"
  ],
  "modality": "text",
  "selected_attempt_id": "T10-attempt-00",
  "selected_attempt_index": 0,
  "evaluated_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/T10-attempt-00/asset.txt",
    "sha256": "cde565a898ffffbca2c7f184efa9cf5dad3d8ceb889e680ca28b71731996a2cc",
    "bytes": 469
  },
  "native_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/T10-attempt-00/asset.txt",
    "sha256": "cde565a898ffffbca2c7f184efa9cf5dad3d8ceb889e680ca28b71731996a2cc",
    "bytes": 469
  },
  "acceptance_receipt": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/T10-attempt-00/receipt.json",
    "sha256": "0aa2d45ab1b4bb5f44dc5ef5eb9d5a1ca1a9ef0b8383f7750766519a9f242b86",
    "bytes": 557
  },
  "generator_id": null,
  "generation_method": "Original fictional Japanese email authored before predictions; no generative model used.",
  "seed": null,
  "settings": null
}
  

元の動画5本:危険な場面が含まれるか

動画の中に、自車がすぐに対応する必要のある危険な場面が映っているかを質問しました。その後に回避できても、途中に危険があればDangerousとする基準です。

V01は子どもが車の進路に入る動画、V02は別の車が前方へ入る動画です。この2本はDangerous、残る3本はSafeを想定した答えにしました。V01・V02に衝突や負傷は映っていません。

Jev-Omniは、どちらの順番でも5本すべてでSafeを選びました。Dangerousを想定した2本は0/2、Safeを想定した3本は3/3が一致しました。Clefは5本とも一致しました。

ただし、動画には回避や停止まで含まれています。どの時点を見て「危険」と答えるべきかを決めにくく、今回の動画と質問は性能比較に不向きでした。実際の運転で危険を判断できるかは、この結果では分かりません。

V01:子どもが自車の進路へ入る

V01:評価に使用した先頭4秒、96フレーム、24 fps。 元のファイル

想定した答え:Dangerous

子どもが自車の進路に入り、すぐに対応が必要な場面です。

質問と選択肢

モデルに伝えた説明

  Assess only the visible dashcam scene. Dangerous means a visible immediate conflict with the ego vehicle travel path requires prompt braking or evasion. Safe means visible separation, adequate headway, or a stopped ego vehicle removes immediate conflict. The presence of a child, rain, or another vehicle alone does not establish danger.
  

質問

  Does any interval in this clip show an immediate dangerous conflict with the ego vehicle travel path, or is the situation safe throughout? Later successful braking does not erase an earlier dangerous conflict.
  

元の順番

  ["Dangerous", "Safe"]
  

逆順

  ["Safe", "Dangerous"]
  
モデル元の順番の回答逆順の回答想定回答との一致:元 / 逆
Jev-OmniSafeSafe不一致 / 不一致
ClefDangerousDangerous一致 / 一致
選択肢Jev-Omni:元Jev-Omni:逆Clef:元Clef:逆
Dangerous0.1311730.4373430.9167080.934395
Safe0.8688270.5626570.0832920.065605
保存済み回答と実験ID
生成プロンプトと設定
  Four-second synthetic realistic forward-facing dashcam clip, fixed camera inside ego car, coherent continuous shot, no cuts, no text overlays, labels, captions, logos or watermarks. Physically consistent vehicles, road geometry and actor motion. Non-graphic: no injury, no collision, no impact. On a quiet residential street in daylight, ego car moves steadily forward at a plausible moderate speed. A clearly visible fictional child on the right sidewalk approaches the curb from 0.4 seconds. At 1.6 seconds the child steps from the curb into the ego lane a short distance ahead, continuing toward the lane center. Show converging paths and rapidly shrinking stopping gap by 2.5 seconds; ego brakes promptly. End while separated before impact.
  
  {
  "fixture_id": "V01",
  "cohorts": [
    "E1_JEV_INITIAL",
    "E2_CLEF_COMMON",
    "E3_EMERGENCY_ORIGINAL"
  ],
  "modality": "video",
  "selected_attempt_id": "V01-attempt-00",
  "selected_attempt_index": 0,
  "evaluated_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/V01-attempt-00/asset.mp4",
    "sha256": "dcd7a01b8b25b24b46df0a9af7508a5b2443bc200225bdd9df4587a241f56014",
    "bytes": 3691855
  },
  "native_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/V01-attempt-00/native.mp4",
    "sha256": "2d902c8c5e945b74f787ffdcae22d2406cb0a2d8051c0e7bad527b0d72a81450",
    "bytes": 2577780
  },
  "acceptance_receipt": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/V01-attempt-00/receipt.json",
    "sha256": "9148fd6d0fef2f3eac570f287fcda50ccf838c9d4060ddc74179c41085d05da9",
    "bytes": 3576
  },
  "observed_media": {
    "native": {
      "sha256": "2d902c8c5e945b74f787ffdcae22d2406cb0a2d8051c0e7bad527b0d72a81450",
      "duration_seconds": 4.458333,
      "width": 1344,
      "height": 768,
      "fps": "24/1",
      "codec": "h264",
      "frames": 107,
      "bytes": 2577780
    },
    "evaluated_transport": {
      "sha256": "dcd7a01b8b25b24b46df0a9af7508a5b2443bc200225bdd9df4587a241f56014",
      "bytes": 3691855,
      "duration_seconds": 4,
      "frames": 96,
      "fps": 24
    }
  },
  "generator_id": "minimax_h3_pdd8",
  "prompt_field": "prompt",
  "seed": 2026100201,
  "seed_status": "requested; no cross-hardware determinism claimed",
  "submitted_settings": {
    "seconds": 4,
    "aspect_ratio": "16:9",
    "seed": 2026100201
  },
  "fixture_or_configured_settings": {
    "format": "MP4",
    "seed": 2026100201,
    "width": 1344,
    "height": 768,
    "duration_seconds": 4,
    "fps": 24,
    "api_steps": 9,
    "short_edge": 768,
    "aspect_ratio": "16:9",
    "flow_shift_high": 12,
    "flow_shift_low": 3
  },
  "generation_request": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/V01-attempt-00/generation.request.json",
    "sha256": "6751480e8f8b19faa6579718fd1920fc9b55cad49bc752dfe947de109f7928a9",
    "bytes": 850
  },
  "recorded_runtime_identity_source": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/V01-attempt-00/generation.json",
    "sha256": "5914ef666fdbded464bdda47e5ed043dcf01db66f9d4de839f39087fcc7fa662",
    "bytes": 103820
  },
  "request_vs_configuration_notice": "Submitted settings and recorded runtime/fixture settings are separate evidence; a prepared generation record alone is not proof of execution. The selected asset and pre-inference acceptance receipt establish the retained input."
}
  

V02:前方へ車が割り込む

V02:評価に使用した先頭4秒、96フレーム、24 fps。 元のファイル

想定した答え:Dangerous

別の車が自車の進路に入り、すぐに対応が必要な場面です。

質問と選択肢

モデルに伝えた説明

  Assess only the visible dashcam scene. Dangerous means a visible immediate conflict with the ego vehicle travel path requires prompt braking or evasion. Safe means visible separation, adequate headway, or a stopped ego vehicle removes immediate conflict. The presence of a child, rain, or another vehicle alone does not establish danger.
  

質問

  Does any interval in this clip show an immediate dangerous conflict with the ego vehicle travel path, or is the situation safe throughout? Later successful braking does not erase an earlier dangerous conflict.
  

元の順番

  ["Dangerous", "Safe"]
  

逆順

  ["Safe", "Dangerous"]
  
モデル元の順番の回答逆順の回答想定回答との一致:元 / 逆
Jev-OmniSafeSafe不一致 / 不一致
ClefDangerousDangerous一致 / 一致
選択肢Jev-Omni:元Jev-Omni:逆Clef:元Clef:逆
Dangerous0.1104710.4215520.5515740.661048
Safe0.8895290.5784480.4484260.338952
保存済み回答と実験ID
生成プロンプトと設定
  Four-second synthetic realistic forward-facing dashcam clip, fixed camera inside ego car, coherent continuous shot, no cuts, no text overlays, labels, captions, logos or watermarks. Physically consistent vehicles, road geometry and actor motion. Non-graphic: no injury, no collision, no impact. On a daylight urban two-lane road, ego moves forward behind open road. A car in the adjacent right lane begins a clear leftward move at 0.8 seconds. At 1.4 seconds it cuts across the lane marking into the ego lane directly ahead with very short following distance; show rear vehicle size increasing and a closing gap. Ego brakes promptly. End before contact.
  
  {
  "fixture_id": "V02",
  "cohorts": [
    "E1_JEV_INITIAL",
    "E2_CLEF_COMMON",
    "E3_EMERGENCY_ORIGINAL"
  ],
  "modality": "video",
  "selected_attempt_id": "V02-attempt-00",
  "selected_attempt_index": 0,
  "evaluated_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/V02-attempt-00/asset.mp4",
    "sha256": "4b23e1598cf11faf1711d2416d21b93531a38222dd23ee5d797290c3ba6cb96e",
    "bytes": 2340360
  },
  "native_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/V02-attempt-00/native.mp4",
    "sha256": "ea0bba64e4a7ae24d6ce6016847b2d2ade4c49e5d2c8c08f5d5c1b8923cae12d",
    "bytes": 1892973
  },
  "acceptance_receipt": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/V02-attempt-00/receipt.json",
    "sha256": "2d4be3befc00dc0213186394d80f59436a7fdf7602be31162a046cbc52d8866d",
    "bytes": 3509
  },
  "observed_media": {
    "native": {
      "sha256": "ea0bba64e4a7ae24d6ce6016847b2d2ade4c49e5d2c8c08f5d5c1b8923cae12d",
      "duration_seconds": 4.458333,
      "width": 1344,
      "height": 768,
      "fps": "24/1",
      "codec": "h264",
      "frames": 107,
      "bytes": 1892973
    },
    "evaluated_transport": {
      "sha256": "4b23e1598cf11faf1711d2416d21b93531a38222dd23ee5d797290c3ba6cb96e",
      "bytes": 2340360,
      "duration_seconds": 4,
      "frames": 96,
      "fps": 24
    }
  },
  "generator_id": "minimax_h3_pdd8",
  "prompt_field": "prompt",
  "seed": 2026100202,
  "seed_status": "requested; no cross-hardware determinism claimed",
  "submitted_settings": {
    "seconds": 4,
    "aspect_ratio": "16:9",
    "seed": 2026100202
  },
  "fixture_or_configured_settings": {
    "format": "MP4",
    "seed": 2026100202,
    "width": 1344,
    "height": 768,
    "duration_seconds": 4,
    "fps": 24,
    "api_steps": 9,
    "short_edge": 768,
    "aspect_ratio": "16:9",
    "flow_shift_high": 12,
    "flow_shift_low": 3
  },
  "generation_request": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/V02-attempt-00/generation.request.json",
    "sha256": "c595d09476da1b19da583fcd5de702639e214134f56c199d0b2937a307a923cb",
    "bytes": 759
  },
  "recorded_runtime_identity_source": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/V02-attempt-00/generation.json",
    "sha256": "aa5568995b80bee66ac6c44eefdfa63c7f5608b4b490c91d1f4bdaf73f4e8bb3",
    "bytes": 103676
  },
  "request_vs_configuration_notice": "Submitted settings and recorded runtime/fixture settings are separate evidence; a prepared generation record alone is not proof of execution. The selected asset and pre-inference acceptance receipt establish the retained input."
}
  

V08:距離を保って前を走る車

V08:評価に使用した先頭4秒、96フレーム、24 fps。 元のファイル

想定した答え:Safe

車間があり、すぐに対応が必要な危険は見えません。

質問と選択肢

モデルに伝えた説明

  Assess only the visible dashcam scene. Dangerous means a visible immediate conflict with the ego vehicle travel path requires prompt braking or evasion. Safe means visible separation, adequate headway, or a stopped ego vehicle removes immediate conflict. The presence of a child, rain, or another vehicle alone does not establish danger.
  

質問

  Does any interval in this clip show an immediate dangerous conflict with the ego vehicle travel path, or is the situation safe throughout? Later successful braking does not erase an earlier dangerous conflict.
  

元の順番

  ["Dangerous", "Safe"]
  

逆順

  ["Safe", "Dangerous"]
  
モデル元の順番の回答逆順の回答想定回答との一致:元 / 逆
Jev-OmniSafeSafe一致 / 一致
ClefSafeSafe一致 / 一致
選択肢Jev-Omni:元Jev-Omni:逆Clef:元Clef:逆
Dangerous0.0710740.3259500.0089160.008413
Safe0.9289260.6740500.9910840.991587
保存済み回答と実験ID
生成プロンプトと設定
  Four-second synthetic realistic forward-facing dashcam clip, fixed camera inside ego car, coherent continuous shot, no cuts, no text overlays, labels, captions, logos or watermarks. Physically consistent vehicles, road geometry and actor motion. Non-graphic: no injury, no collision, no impact. Daylight dashcam on a straight road. Ego follows one car well ahead with clearly distant, nonclosing separation and substantial road visible between ego and the leading car at their modest steady speed. Both continue straight, leading car maintains nearly constant apparent size and relative position throughout all four seconds. Open road, no braking, no lane changes, no pedestrians.
  
  {
  "fixture_id": "V08",
  "cohorts": [
    "E1_JEV_INITIAL",
    "E2_CLEF_COMMON",
    "E3_EMERGENCY_ORIGINAL"
  ],
  "modality": "video",
  "selected_attempt_id": "V08-attempt-00",
  "selected_attempt_index": 0,
  "evaluated_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/V08-attempt-00/asset.mp4",
    "sha256": "0522b5fc81629cb5dcc242a177d955feebe15f4b1805e17f952f9741c20e7f92",
    "bytes": 1512796
  },
  "native_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/V08-attempt-00/native.mp4",
    "sha256": "908544e0b7c40d48f741a03c6806d12d599151894e2c1d872c73343a2596e6e4",
    "bytes": 1145176
  },
  "acceptance_receipt": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/V08-attempt-00/receipt.json",
    "sha256": "8dee295628be4cf88d390d6cf7bd4f5938681a290b4040cace4c5958492acd66",
    "bytes": 3184
  },
  "observed_media": {
    "native": {
      "sha256": "908544e0b7c40d48f741a03c6806d12d599151894e2c1d872c73343a2596e6e4",
      "duration_seconds": 4.458333,
      "width": 1344,
      "height": 768,
      "fps": "24/1",
      "codec": "h264",
      "frames": 107,
      "bytes": 1145176
    },
    "evaluated_transport": {
      "sha256": "0522b5fc81629cb5dcc242a177d955feebe15f4b1805e17f952f9741c20e7f92",
      "bytes": 1512796,
      "duration_seconds": 4,
      "frames": 96,
      "fps": 24
    }
  },
  "generator_id": "minimax_h3_pdd8",
  "prompt_field": "prompt",
  "seed": 2026100208,
  "seed_status": "requested; no cross-hardware determinism claimed",
  "submitted_settings": {
    "seconds": 4,
    "aspect_ratio": "16:9",
    "seed": 2026100208
  },
  "fixture_or_configured_settings": {
    "format": "MP4",
    "seed": 2026100208,
    "width": 1344,
    "height": 768,
    "duration_seconds": 4,
    "fps": 24,
    "api_steps": 9,
    "short_edge": 768,
    "aspect_ratio": "16:9",
    "flow_shift_high": 12,
    "flow_shift_low": 3
  },
  "generation_request": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/V08-attempt-00/generation.request.json",
    "sha256": "7da0f7da23db7d5286c724e892e8bf9d1e58118d533d80ce8845ccc57fb3c6e1",
    "bytes": 786
  },
  "recorded_runtime_identity_source": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/V08-attempt-00/generation.json",
    "sha256": "2fe9db64ddae1a33bfc7780f487e7971392e78b13231887e4d5af210ea7efc04",
    "bytes": 103604
  },
  "request_vs_configuration_notice": "Submitted settings and recorded runtime/fixture settings are separate evidence; a prepared generation record alone is not proof of execution. The selected asset and pre-inference acceptance receipt establish the retained input."
}
  

V09:分離帯で隔てられた対向車

V09:評価に使用した先頭4秒、96フレーム、24 fps。 元のファイル

想定した答え:Safe

対向車との間には分離帯があり、自車の進路に入るものはありません。

質問と選択肢

モデルに伝えた説明

  Assess only the visible dashcam scene. Dangerous means a visible immediate conflict with the ego vehicle travel path requires prompt braking or evasion. Safe means visible separation, adequate headway, or a stopped ego vehicle removes immediate conflict. The presence of a child, rain, or another vehicle alone does not establish danger.
  

質問

  Does any interval in this clip show an immediate dangerous conflict with the ego vehicle travel path, or is the situation safe throughout? Later successful braking does not erase an earlier dangerous conflict.
  

元の順番

  ["Dangerous", "Safe"]
  

逆順

  ["Safe", "Dangerous"]
  
モデル元の順番の回答逆順の回答想定回答との一致:元 / 逆
Jev-OmniSafeSafe一致 / 一致
ClefSafeSafe一致 / 一致
選択肢Jev-Omni:元Jev-Omni:逆Clef:元Clef:逆
Dangerous0.0399370.3023910.0072330.006641
Safe0.9600630.6976090.9927670.993359
保存済み回答と実験ID
生成プロンプトと設定
  Four-second synthetic realistic forward-facing dashcam clip, fixed camera inside ego car, coherent continuous shot, no cuts, no text overlays, labels, captions, logos or watermarks. Physically consistent vehicles, road geometry and actor motion. Non-graphic: no injury, no collision, no impact. Daylight dashcam in light rain, with small droplets near windshield edges but a clear central view. Ego moves steadily at a modest speed along an empty lane on a divided road with a continuous central barrier. Show generous forward sight distance, no standing water, no skidding, and no vehicles or people entering the lane for all four seconds.
  
  {
  "fixture_id": "V09",
  "cohorts": [
    "E1_JEV_INITIAL",
    "E2_CLEF_COMMON",
    "E3_EMERGENCY_ORIGINAL"
  ],
  "modality": "video",
  "selected_attempt_id": "V09-attempt-01",
  "selected_attempt_index": 1,
  "evaluated_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/V09-attempt-01/asset.mp4",
    "sha256": "cdcce5e0b09700ed8b457ea81e5415c86720a889b47237d394e89276cfe72330",
    "bytes": 1852141
  },
  "native_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/V09-attempt-01/native.mp4",
    "sha256": "92e1afd4175c487dec865e0d32b90617e0e93e12e585d0530c93722f9ef0d17e",
    "bytes": 1488990
  },
  "acceptance_receipt": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/V09-attempt-01/receipt.json",
    "sha256": "816a71502951f3d06ee034afffd46b34743493dd89ecd884270559a9c6ebe9bc",
    "bytes": 3624
  },
  "observed_media": {
    "native": {
      "sha256": "92e1afd4175c487dec865e0d32b90617e0e93e12e585d0530c93722f9ef0d17e",
      "duration_seconds": 4.458333,
      "width": 1344,
      "height": 768,
      "fps": "24/1",
      "codec": "h264",
      "frames": 107,
      "bytes": 1488990
    },
    "evaluated_transport": {
      "sha256": "cdcce5e0b09700ed8b457ea81e5415c86720a889b47237d394e89276cfe72330",
      "bytes": 1852141,
      "duration_seconds": 4,
      "frames": 96,
      "fps": 24
    }
  },
  "generator_id": "minimax_h3_pdd8",
  "prompt_field": "prompt",
  "seed": 2026101209,
  "seed_status": "requested; no cross-hardware determinism claimed",
  "submitted_settings": {
    "seconds": 4,
    "aspect_ratio": "16:9",
    "seed": 2026101209
  },
  "fixture_or_configured_settings": {
    "format": "MP4",
    "seed": 2026101209,
    "width": 1344,
    "height": 768,
    "duration_seconds": 4,
    "fps": 24,
    "api_steps": 9,
    "short_edge": 768,
    "aspect_ratio": "16:9",
    "flow_shift_high": 12,
    "flow_shift_low": 3
  },
  "generation_request": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/V09-attempt-01/generation.request.json",
    "sha256": "254de09d7582b9de7673f49339d52415789f404de4b6b7f845aa7faa0872c963",
    "bytes": 746
  },
  "recorded_runtime_identity_source": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/V09-attempt-01/generation.json",
    "sha256": "8c5fc59eb20aa084866e5cdfe0449cd0b22a785e7b42bd9563aa92870a50261b",
    "bytes": 103563
  },
  "request_vs_configuration_notice": "Submitted settings and recorded runtime/fixture settings are separate evidence; a prepared generation record alone is not proof of execution. The selected asset and pre-inference acceptance receipt establish the retained input."
}
  

V10:分離された自転車レーン

V10:評価に使用した先頭4秒、96フレーム、24 fps。 元のファイル

想定した答え:Safe

自転車は分離されたレーンにあり、自車の進路には入っていません。

質問と選択肢

モデルに伝えた説明

  Assess only the visible dashcam scene. Dangerous means a visible immediate conflict with the ego vehicle travel path requires prompt braking or evasion. Safe means visible separation, adequate headway, or a stopped ego vehicle removes immediate conflict. The presence of a child, rain, or another vehicle alone does not establish danger.
  

質問

  Does any interval in this clip show an immediate dangerous conflict with the ego vehicle travel path, or is the situation safe throughout? Later successful braking does not erase an earlier dangerous conflict.
  

元の順番

  ["Dangerous", "Safe"]
  

逆順

  ["Safe", "Dangerous"]
  
モデル元の順番の回答逆順の回答想定回答との一致:元 / 逆
Jev-OmniSafeSafe一致 / 一致
ClefSafeSafe一致 / 一致
選択肢Jev-Omni:元Jev-Omni:逆Clef:元Clef:逆
Dangerous0.0430430.2620860.0134280.012920
Safe0.9569570.7379140.9865720.987080
保存済み回答と実験ID
生成プロンプトと設定
  Four-second synthetic realistic forward-facing dashcam clip, fixed camera inside ego car, coherent continuous shot, no cuts, no text overlays, labels, captions, logos or watermarks. Physically consistent vehicles, road geometry and actor motion. Non-graphic: no injury, no collision, no impact. Daylight dashcam with ego moving straight at steady modest speed. A cyclist travels in the same direction to the right, entirely in a protected bike lane separated by a raised continuous curb and regular bollards. Keep the cyclist, curb and bollards clearly visible; cyclist remains behind the separation for all four seconds. Ego lane remains open, no turns or crossings.
  
  {
  "fixture_id": "V10",
  "cohorts": [
    "E1_JEV_INITIAL",
    "E2_CLEF_COMMON",
    "E3_EMERGENCY_ORIGINAL"
  ],
  "modality": "video",
  "selected_attempt_id": "V10-attempt-00",
  "selected_attempt_index": 0,
  "evaluated_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/V10-attempt-00/asset.mp4",
    "sha256": "5cf43c1116b9b095163b8f0df86f4b4865d5c27dae7f926e5e88c6d1dd94457e",
    "bytes": 3608621
  },
  "native_asset": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/V10-attempt-00/native.mp4",
    "sha256": "cc00e7a6bb83ff19d457c576ada2da36be35190bfdfe87029dd0bd68484fc934",
    "bytes": 2992586
  },
  "acceptance_receipt": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/V10-attempt-00/receipt.json",
    "sha256": "ac5999bac89cd93d9a5c5277b51a8f243a98499ecb31ee74e8ef74bce25b4199",
    "bytes": 3318
  },
  "observed_media": {
    "native": {
      "sha256": "cc00e7a6bb83ff19d457c576ada2da36be35190bfdfe87029dd0bd68484fc934",
      "duration_seconds": 4.458333,
      "width": 1344,
      "height": 768,
      "fps": "24/1",
      "codec": "h264",
      "frames": 107,
      "bytes": 2992586
    },
    "evaluated_transport": {
      "sha256": "5cf43c1116b9b095163b8f0df86f4b4865d5c27dae7f926e5e88c6d1dd94457e",
      "bytes": 3608621,
      "duration_seconds": 4,
      "frames": 96,
      "fps": 24
    }
  },
  "generator_id": "minimax_h3_pdd8",
  "prompt_field": "prompt",
  "seed": 2026100210,
  "seed_status": "requested; no cross-hardware determinism claimed",
  "submitted_settings": {
    "seconds": 4,
    "aspect_ratio": "16:9",
    "seed": 2026100210
  },
  "fixture_or_configured_settings": {
    "format": "MP4",
    "seed": 2026100210,
    "width": 1344,
    "height": 768,
    "duration_seconds": 4,
    "fps": 24,
    "api_steps": 9,
    "short_edge": 768,
    "aspect_ratio": "16:9",
    "flow_shift_high": 12,
    "flow_shift_low": 3
  },
  "generation_request": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/V10-attempt-00/generation.request.json",
    "sha256": "9f06d2a26666eefcd0837995375217486b70cb0201bc707f47f745257750b16d",
    "bytes": 773
  },
  "recorded_runtime_identity_source": {
    "path": "2026-10-02-jev-omni-40-synthetic/attempts/V10-attempt-00/generation.json",
    "sha256": "bfc3e83254a7327d7743f58a50537632c652ee10283eb1457569fd21655e4bca",
    "bytes": 103605
  },
  "request_vs_configuration_notice": "Submitted settings and recorded runtime/fixture settings are separate evidence; a prepared generation record alone is not proof of execution. The selected asset and pre-inference acceptance receipt establish the retained input."
}
  

元の5本で、緊急停止が必要かを調べました

同じ5本で、衝突を避けるために緊急停止が必要かを質問しました。ここでいう緊急停止は、すぐに強くブレーキをかけて停止することです。通常の減速やハンドル操作だけでは衝突を避けられない場合を指します。

質問だけでなく、説明、判定基準、選択肢も変えました。そのため、回答が変わっても、質問文だけが理由とはいえません。

V01・V02は、速度や残りの距離を映像から十分に読み取れず、緊急停止が必要かを決められませんでした。記録ではambiguousとし、集計から外しています。最初のDangerousという答えは、別の質問に対するものです。

想定した答え動画数各モデルの回答数と扱い
Emergency stop needed00
Emergency stop not needed36回答、すべて一致
ambiguous24回答、正答数の集計から除外

各動画で選択肢の順番を変えて2回ずつ試しました。2つのモデルを合わせると、回答は20件です。動画は前節の5本と同じです。

V01:子どもが自車の進路へ入る

同じ動画を前節で再生する。

想定した答え:映像だけでは決められない(ambiguous)

子どもが進路に入るため、減速や停止は考えられます。ただし、速度や残りの距離が分からず、通常のブレーキで足りるかは決められません。衝突や明確な緊急ブレーキは映っていません。

質問と選択肢

モデルに伝えた説明

  Judge only the visible driving sequence. An emergency stop means immediate, very hard braking intended to bring the ego vehicle to a stop to prevent an imminent collision; ordinary gradual braking or steering alone does not qualify. Consider the whole clip, including hazards later avoided. Do not assume a collision occurred or infer facts hidden outside the video.
  

質問

  At any point in this video, should the ego vehicle make an emergency stop to avoid a collision?
  

元の順番

  ["Emergency stop needed", "Emergency stop not needed"]
  

逆順

  ["Emergency stop not needed", "Emergency stop needed"]
  
モデル元の順番の回答逆順の回答想定回答との一致:元 / 逆
Jev-OmniEmergency stop not neededEmergency stop not needed除外 / 除外
ClefEmergency stop not neededEmergency stop not needed除外 / 除外
選択肢Jev-Omni:元Jev-Omni:逆Clef:元Clef:逆
Emergency stop needed0.0460340.3571190.1738490.346873
Emergency stop not needed0.9539660.6428810.8261510.653127
保存済み回答と実験ID

V02:前方へ車が割り込む

同じ動画を前節で再生する。

想定した答え:映像だけでは決められない(ambiguous)

右前方の車が徐々に自車の進路に入り、減速や車間の確保が必要になります。速度や距離が分からず、緊急停止が必要かは決められません。衝突は映っていません。

質問と選択肢

モデルに伝えた説明

  Judge only the visible driving sequence. An emergency stop means immediate, very hard braking intended to bring the ego vehicle to a stop to prevent an imminent collision; ordinary gradual braking or steering alone does not qualify. Consider the whole clip, including hazards later avoided. Do not assume a collision occurred or infer facts hidden outside the video.
  

質問

  At any point in this video, should the ego vehicle make an emergency stop to avoid a collision?
  

元の順番

  ["Emergency stop needed", "Emergency stop not needed"]
  

逆順

  ["Emergency stop not needed", "Emergency stop needed"]
  
モデル元の順番の回答逆順の回答想定回答との一致:元 / 逆
Jev-OmniEmergency stop not neededEmergency stop not needed除外 / 除外
ClefEmergency stop not neededEmergency stop not needed除外 / 除外
選択肢Jev-Omni:元Jev-Omni:逆Clef:元Clef:逆
Emergency stop needed0.0120540.1578960.0329610.076961
Emergency stop not needed0.9879460.8421040.9670390.923039
保存済み回答と実験ID

V08:距離を保って前を走る車

同じ動画を前節で再生する。

想定した答え:Emergency stop not needed

前の車は車線内にあり、自車の進路は開いています。急な割り込みや障害物はなく、緊急停止が必要な場面は見えません。車間の調整が不要という意味ではありません。

質問と選択肢

モデルに伝えた説明

  Judge only the visible driving sequence. An emergency stop means immediate, very hard braking intended to bring the ego vehicle to a stop to prevent an imminent collision; ordinary gradual braking or steering alone does not qualify. Consider the whole clip, including hazards later avoided. Do not assume a collision occurred or infer facts hidden outside the video.
  

質問

  At any point in this video, should the ego vehicle make an emergency stop to avoid a collision?
  

元の順番

  ["Emergency stop needed", "Emergency stop not needed"]
  

逆順

  ["Emergency stop not needed", "Emergency stop needed"]
  
モデル元の順番の回答逆順の回答想定回答との一致:元 / 逆
Jev-OmniEmergency stop not neededEmergency stop not needed一致 / 一致
ClefEmergency stop not neededEmergency stop not needed一致 / 一致
選択肢Jev-Omni:元Jev-Omni:逆Clef:元Clef:逆
Emergency stop needed0.0132230.1385200.0060040.006004
Emergency stop not needed0.9867770.8614800.9939960.993996
保存済み回答と実験ID

V09:分離帯で隔てられた対向車

同じ動画を前節で再生する。

想定した答え:Emergency stop not needed

対向車はコンクリートの分離帯の向こうにあり、自車の進路には入っていません。濡れた路面や水滴だけでは、緊急停止が必要とはしていません。

質問と選択肢

モデルに伝えた説明

  Judge only the visible driving sequence. An emergency stop means immediate, very hard braking intended to bring the ego vehicle to a stop to prevent an imminent collision; ordinary gradual braking or steering alone does not qualify. Consider the whole clip, including hazards later avoided. Do not assume a collision occurred or infer facts hidden outside the video.
  

質問

  At any point in this video, should the ego vehicle make an emergency stop to avoid a collision?
  

元の順番

  ["Emergency stop needed", "Emergency stop not needed"]
  

逆順

  ["Emergency stop not needed", "Emergency stop needed"]
  
モデル元の順番の回答逆順の回答想定回答との一致:元 / 逆
Jev-OmniEmergency stop not neededEmergency stop not needed一致 / 一致
ClefEmergency stop not neededEmergency stop not needed一致 / 一致
選択肢Jev-Omni:元Jev-Omni:逆Clef:元Clef:逆
Emergency stop needed0.0057300.0897730.0051000.004982
Emergency stop not needed0.9942700.9102270.9949000.995018
保存済み回答と実験ID

V10:分離された自転車レーン

同じ動画を前節で再生する。

想定した答え:Emergency stop not needed

自転車は段差やポールで隔てられたレーンにとどまり、自車の進路は開いています。交差点での通常の減速や停止とは分けて判断しています。

質問と選択肢

モデルに伝えた説明

  Judge only the visible driving sequence. An emergency stop means immediate, very hard braking intended to bring the ego vehicle to a stop to prevent an imminent collision; ordinary gradual braking or steering alone does not qualify. Consider the whole clip, including hazards later avoided. Do not assume a collision occurred or infer facts hidden outside the video.
  

質問

  At any point in this video, should the ego vehicle make an emergency stop to avoid a collision?
  

元の順番

  ["Emergency stop needed", "Emergency stop not needed"]
  

逆順

  ["Emergency stop not needed", "Emergency stop needed"]
  
モデル元の順番の回答逆順の回答想定回答との一致:元 / 逆
Jev-OmniEmergency stop not neededEmergency stop not needed一致 / 一致
ClefEmergency stop not neededEmergency stop not needed一致 / 一致
選択肢Jev-Omni:元Jev-Omni:逆Clef:元Clef:逆
Emergency stop needed0.0067980.0877200.0072890.007877
Emergency stop not needed0.9932030.9122800.9927110.992123
保存済み回答と実験ID

接近する場面の動画5本を追加しました

接触する場面を5種類考え、各2回、計10回生成しました。使う条件は、4秒の中に接近と明確な接触の両方が映っていることです。近づくだけの映像や、ボンネットで相手が隠れる映像は、接触を確認できたとは扱いませんでした。

10回とも条件を満たさず、使える接触動画は0本、接触動画としての分類も0回でした。そこで条件を変更し、各ケースで最後に生成したC01〜C05を、接近する場面として使いました。動画は作り直していません。

C01〜C04は映像だけでは答えを決められず、ambiguousとして集計から外しました。C05は「緊急停止は不要」としました。各モデルはC05の2回答とも一致し、C01〜C04の8回答は集計から外しています。

ClefはC02・C03で答えが変わりました

選択肢を逆の順番にすると、ClefはC02・C03で「緊急停止は不要」から「緊急停止が必要」に変わりました。Jev-Omniの答えは変わりませんでしたが、確率は変わりました。

モデル / 動画「必要」の確率:元の順番「必要」の確率:逆順
Clef / C020.3895470.609524
Clef / C030.3979380.577495
Jev-Omni / C020.0083160.140749
Jev-Omni / C030.0177120.241847

この2本には想定した答えがないため、回答の変化を正解や不正解として数えていません。

C01:赤いセダンへの接近

C01:評価に使用した先頭4秒、96フレーム、24 fps。 元のファイル

想定した答え:映像だけでは決められない(ambiguous)

車間が縮まり、後部ランプも見えるため、減速は考えられます。ただし、接近する速度や停止までの距離が分からず、緊急停止が必要かは決められません。最後に近づいたことだけで、衝突とは判断していません。

質問と選択肢

モデルに伝えた説明

  Judge only the visible driving sequence. An emergency stop means immediate, very hard braking intended to bring the ego vehicle to a stop to prevent an imminent collision; ordinary gradual braking or steering alone does not qualify. Consider the whole clip, including hazards later avoided. Do not assume a collision occurred or infer facts hidden outside the video.
  

質問

  At any point in this video, should the ego vehicle make an emergency stop to avoid a collision?
  

元の順番

  ["Emergency stop needed", "Emergency stop not needed"]
  

逆順

  ["Emergency stop not needed", "Emergency stop needed"]
  
モデル元の順番の回答逆順の回答想定回答との一致:元 / 逆
Jev-OmniEmergency stop not neededEmergency stop not needed除外 / 除外
ClefEmergency stop not neededEmergency stop not needed除外 / 除外
選択肢Jev-Omni:元Jev-Omni:逆Clef:元Clef:逆
Emergency stop needed0.0160290.1787410.1772410.405449
Emergency stop not needed0.9839710.8212590.8227590.594551
保存済み回答と実験ID
生成プロンプトと設定
  Four-second synthetic realistic forward-facing dashcam video, a fixed camera inside the ego vehicle, one continuous physically coherent shot, no cuts. Daylight on a closed road used only by empty fictional vehicles. No people, pedestrians, cyclists, visible occupants, injuries, blood, fire or graphic damage. No captions, text, labels, logos or watermarks. Show uninterrupted ego forward motion, visible road markings and changing vehicle separation before directly visible non-graphic vehicle-to-vehicle contact. A red sedan is stopped in the middle of the ego lane on a straight level road. It is visible directly ahead from the first frame. The ego vehicle continues moving forward, the stopped sedan grows rapidly larger, and the gap closes continuously from 0.3 to 2.5 seconds. A clear pre-impact interval shows the blocked route and urgent opportunity to brake to a stop. At about 3.0 seconds the ego front bumper visibly contacts the sedan rear bumper; the sedan body jolts forward slightly while both intact bumpers visibly meet. Keep the scene continuous and contact visible before 3.5 seconds.
  
  {
  "fixture_id": "C01",
  "cohorts": [
    "E4_NEAR_APPROACH_AMENDMENT"
  ],
  "modality": "video",
  "selected_attempt_id": "C01.attempt-1",
  "selected_attempt_index": 1,
  "generator_id": "minimax_h3_pdd8",
  "prompt_field": "prompt",
  "seed": 2026101401,
  "seed_status": "requested; no cross-hardware determinism claimed",
  "submitted_settings": {
    "seconds": 4,
    "aspect_ratio": "16:9",
    "seed": 2026101401
  },
  "configured_request_controls": {
    "num_inference_steps": 9,
    "flow_shift": 12.0,
    "audio_flow_shift": 3.0,
    "target": {
      "short_edge": 768,
      "aspect_ratio": "16:9"
    },
    "duration_seconds": 4,
    "fps": 24
  },
  "generation_request": {
    "path": "2026-10-02-clef-jev-collision/C01.attempt-1.generation.request.json",
    "sha256": "46f8aae33f19a8fcdc691864f7c247918a2f5d5d0ca502c65362e629c5d50cf4",
    "bytes": 1223
  },
  "recorded_runtime_identity_source": {
    "path": "2026-10-02-clef-jev-collision/h3-loaded-identity.json",
    "sha256": "8682073cf08b2dde8ac45bd592c47a9a1aaede7c5e9e577804a7e6eb003f11be",
    "bytes": 4104
  },
  "native_asset": {
    "path": "2026-10-02-clef-jev-collision/attempts/C01.attempt-1/native.mp4",
    "sha256": "e5e24e4e27ac1d22034c821d730701612f81a831cdebfb45c2dda3619f7b3d6a",
    "bytes": 1589923
  },
  "evaluated_asset": {
    "path": "2026-10-02-clef-jev-near-approach/inputs/C01/asset.mp4",
    "sha256": "bf1bb41e1947614f6bb59ffedc492696177bc79220b6aabe76206cf7dafa764b",
    "bytes": 2340198
  },
  "observed_media": {
    "native_frame_count": 107,
    "native_dimensions": null,
    "native_dimensions_notice": "Selected final inspection does not record native dimensions; configured target is 1344x768. No new probe was run.",
    "evaluated_transport_frames": 96,
    "evaluated_transport_fps": 24,
    "evaluated_transport_duration_seconds": 4
  },
  "inspection": {
    "path": "2026-10-02-clef-jev-near-approach/references/C01.attempt-1.final-inspected.json",
    "sha256": "a22d2da04ddf086bebd7a39dcc0b2dfc97dafd80b2cd5c5b0138ee68d43305dd",
    "bytes": 5392
  },
  "reference_classification": "ambiguous",
  "original_collision_media_accepted": false,
  "selection_notice": "Selected final attempt index 1 for each slot under a separately frozen near-approach amendment; no result-based selection. The original prompt requested contact, but visible contact was not independently confirmed."
}
  

C02:白いバンへの接近

C02:評価に使用した先頭4秒、96フレーム、24 fps。 元のファイル

想定した答え:映像だけでは決められない(ambiguous)

バンとの間隔は縮まっていますが、速度や停止までの距離が分かりません。通常の減速で足りるか、緊急停止が必要かは決められません。最後の位置や陰影だけで、衝突とは判断していません。

質問と選択肢

モデルに伝えた説明

  Judge only the visible driving sequence. An emergency stop means immediate, very hard braking intended to bring the ego vehicle to a stop to prevent an imminent collision; ordinary gradual braking or steering alone does not qualify. Consider the whole clip, including hazards later avoided. Do not assume a collision occurred or infer facts hidden outside the video.
  

質問

  At any point in this video, should the ego vehicle make an emergency stop to avoid a collision?
  

元の順番

  ["Emergency stop needed", "Emergency stop not needed"]
  

逆順

  ["Emergency stop not needed", "Emergency stop needed"]
  
モデル元の順番の回答逆順の回答想定回答との一致:元 / 逆
Jev-OmniEmergency stop not neededEmergency stop not needed除外 / 除外
ClefEmergency stop not neededEmergency stop needed除外 / 除外
選択肢Jev-Omni:元Jev-Omni:逆Clef:元Clef:逆
Emergency stop needed0.0083160.1407490.3895470.609524
Emergency stop not needed0.9916840.8592510.6104530.390476
保存済み回答と実験ID
生成プロンプトと設定
  Four-second synthetic realistic forward-facing dashcam video, a fixed camera inside the ego vehicle, one continuous physically coherent shot, no cuts. Daylight on a closed road used only by empty fictional vehicles. No people, pedestrians, cyclists, visible occupants, injuries, blood, fire or graphic damage. No captions, text, labels, logos or watermarks. Show uninterrupted ego forward motion, visible road markings and changing vehicle separation before directly visible non-graphic vehicle-to-vehicle contact. An empty white van travels in the ego lane directly ahead from the first frame. At 0.4 seconds its brake lights illuminate and it rapidly slows to a stop. The ego vehicle keeps advancing, so the van enlarges and the visible following gap shrinks quickly over the next two seconds. Show a clear pre-impact interval during which hard braking to a stop is available and necessary to avoid the stationary van. At about 3.1 seconds the ego front bumper visibly contacts the van rear bumper and the van shifts forward slightly, with both vehicles retaining coherent bodies. All approach and contact are visible before 3.5 seconds.
  
  {
  "fixture_id": "C02",
  "cohorts": [
    "E4_NEAR_APPROACH_AMENDMENT"
  ],
  "modality": "video",
  "selected_attempt_id": "C02.attempt-1",
  "selected_attempt_index": 1,
  "generator_id": "minimax_h3_pdd8",
  "prompt_field": "prompt",
  "seed": 2026101402,
  "seed_status": "requested; no cross-hardware determinism claimed",
  "submitted_settings": {
    "seconds": 4,
    "aspect_ratio": "16:9",
    "seed": 2026101402
  },
  "configured_request_controls": {
    "num_inference_steps": 9,
    "flow_shift": 12.0,
    "audio_flow_shift": 3.0,
    "target": {
      "short_edge": 768,
      "aspect_ratio": "16:9"
    },
    "duration_seconds": 4,
    "fps": 24
  },
  "generation_request": {
    "path": "2026-10-02-clef-jev-collision/C02.attempt-1.generation.request.json",
    "sha256": "4d6e628c5ec435523792444799caee6c83eb25340a20a76a3cdf6aec3ba68618",
    "bytes": 1258
  },
  "recorded_runtime_identity_source": {
    "path": "2026-10-02-clef-jev-collision/h3-loaded-identity.json",
    "sha256": "8682073cf08b2dde8ac45bd592c47a9a1aaede7c5e9e577804a7e6eb003f11be",
    "bytes": 4104
  },
  "native_asset": {
    "path": "2026-10-02-clef-jev-collision/attempts/C02.attempt-1/native.mp4",
    "sha256": "4c3661cc6e148509355e852519dba702cd7bf5bd2d903bcae71c35436bd06917",
    "bytes": 1627265
  },
  "evaluated_asset": {
    "path": "2026-10-02-clef-jev-near-approach/inputs/C02/asset.mp4",
    "sha256": "ebebbf186ff2f12392950c6bcd0c96de9512af5cc8d346a0999903d34499155c",
    "bytes": 2366486
  },
  "observed_media": {
    "native_frame_count": 107,
    "native_dimensions": null,
    "native_dimensions_notice": "Selected final inspection does not record native dimensions; configured target is 1344x768. No new probe was run.",
    "evaluated_transport_frames": 96,
    "evaluated_transport_fps": 24,
    "evaluated_transport_duration_seconds": 4
  },
  "inspection": {
    "path": "2026-10-02-clef-jev-near-approach/references/C02.attempt-1.final-inspected.json",
    "sha256": "746c3e825014a9435051ec38e4be317d8d9681d6fb9bcff2f7e924fdf9290dd1",
    "bytes": 5392
  },
  "reference_classification": "ambiguous",
  "original_collision_media_accepted": false,
  "selection_notice": "Selected final attempt index 1 for each slot under a separately frozen near-approach amendment; no result-based selection. The original prompt requested contact, but visible contact was not independently confirmed."
}
  

C03:進路を横切るピックアップトラック

C03:評価に使用した先頭4秒、96フレーム、24 fps。 元のファイル

想定した答え:映像だけでは決められない(ambiguous)

トラックが進路を横切っています。ただし、速度や停止距離が分からず、通常の減速やハンドル操作で足りるかは決められません。

質問と選択肢

モデルに伝えた説明

  Judge only the visible driving sequence. An emergency stop means immediate, very hard braking intended to bring the ego vehicle to a stop to prevent an imminent collision; ordinary gradual braking or steering alone does not qualify. Consider the whole clip, including hazards later avoided. Do not assume a collision occurred or infer facts hidden outside the video.
  

質問

  At any point in this video, should the ego vehicle make an emergency stop to avoid a collision?
  

元の順番

  ["Emergency stop needed", "Emergency stop not needed"]
  

逆順

  ["Emergency stop not needed", "Emergency stop needed"]
  
モデル元の順番の回答逆順の回答想定回答との一致:元 / 逆
Jev-OmniEmergency stop not neededEmergency stop not needed除外 / 除外
ClefEmergency stop not neededEmergency stop needed除外 / 除外
選択肢Jev-Omni:元Jev-Omni:逆Clef:元Clef:逆
Emergency stop needed0.0177120.2418470.3979380.577495
Emergency stop not needed0.9822880.7581530.6020620.422505
保存済み回答と実験ID
生成プロンプトと設定
  Four-second synthetic realistic forward-facing dashcam video, a fixed camera inside the ego vehicle, one continuous physically coherent shot, no cuts. Daylight on a closed road used only by empty fictional vehicles. No people, pedestrians, cyclists, visible occupants, injuries, blood, fire or graphic damage. No captions, text, labels, logos or watermarks. Show uninterrupted ego forward motion, visible road markings and changing vehicle separation before directly visible non-graphic vehicle-to-vehicle contact. At a small daylight junction an empty blue pickup is visible ahead on the left from the first frame. From 0.4 to 1.3 seconds it turns across the road into the ego forward lane and then pauses broadside across that lane. The ego vehicle continues forward; the pickup side becomes rapidly larger and its blocking position stays visible for a sustained pre-impact interval, leaving a visible opportunity for very hard braking to a stop. At about 3.1 seconds the ego front bumper visibly contacts the pickup side panel; the pickup rocks or shifts slightly sideways at the contact point. Do not hide contact behind another vehicle or outside the frame. Contact must occur before 3.5 seconds.
  
  {
  "fixture_id": "C03",
  "cohorts": [
    "E4_NEAR_APPROACH_AMENDMENT"
  ],
  "modality": "video",
  "selected_attempt_id": "C03.attempt-1",
  "selected_attempt_index": 1,
  "generator_id": "minimax_h3_pdd8",
  "prompt_field": "prompt",
  "seed": 2026101403,
  "seed_status": "requested; no cross-hardware determinism claimed",
  "submitted_settings": {
    "seconds": 4,
    "aspect_ratio": "16:9",
    "seed": 2026101403
  },
  "configured_request_controls": {
    "num_inference_steps": 9,
    "flow_shift": 12.0,
    "audio_flow_shift": 3.0,
    "target": {
      "short_edge": 768,
      "aspect_ratio": "16:9"
    },
    "duration_seconds": 4,
    "fps": 24
  },
  "generation_request": {
    "path": "2026-10-02-clef-jev-collision/C03.attempt-1.generation.request.json",
    "sha256": "5a99783a84d0c32faaee64a72c35383afda49e627e832bdd8e9a3aeb7995b415",
    "bytes": 1320
  },
  "recorded_runtime_identity_source": {
    "path": "2026-10-02-clef-jev-collision/h3-loaded-identity.json",
    "sha256": "8682073cf08b2dde8ac45bd592c47a9a1aaede7c5e9e577804a7e6eb003f11be",
    "bytes": 4104
  },
  "native_asset": {
    "path": "2026-10-02-clef-jev-collision/attempts/C03.attempt-1/native.mp4",
    "sha256": "c2776962ea247eeaaea6fcccb81f7c0deaa92e88ece6602313282dcbedd07fd6",
    "bytes": 2696603
  },
  "evaluated_asset": {
    "path": "2026-10-02-clef-jev-near-approach/inputs/C03/asset.mp4",
    "sha256": "283c80fc7c43e96537981c374fea22369fd3cfaea6c06f6bbfc23b098ce0e656",
    "bytes": 3819217
  },
  "observed_media": {
    "native_frame_count": 107,
    "native_dimensions": null,
    "native_dimensions_notice": "Selected final inspection does not record native dimensions; configured target is 1344x768. No new probe was run.",
    "evaluated_transport_frames": 96,
    "evaluated_transport_fps": 24,
    "evaluated_transport_duration_seconds": 4
  },
  "inspection": {
    "path": "2026-10-02-clef-jev-near-approach/references/C03.attempt-1.final-inspected.json",
    "sha256": "f177fe1ee9a6c48fa618d806973ca58c2257f686f4e2d363b82aa6242924b295",
    "bytes": 5392
  },
  "reference_classification": "ambiguous",
  "original_collision_media_accepted": false,
  "selection_notice": "Selected final attempt index 1 for each slot under a separately frozen near-approach amendment; no result-based selection. The original prompt requested contact, but visible contact was not independently confirmed."
}
  

C04:右から出てくるバン

C04:評価に使用した先頭4秒、96フレーム、24 fps。 元のファイル

想定した答え:映像だけでは決められない(ambiguous)

バンが右から出てきています。ただし、相対速度や停止距離が分からず、通常の減速やハンドル操作で足りるか、緊急停止が必要かは決められません。接触したとは判断していません。

質問と選択肢

モデルに伝えた説明

  Judge only the visible driving sequence. An emergency stop means immediate, very hard braking intended to bring the ego vehicle to a stop to prevent an imminent collision; ordinary gradual braking or steering alone does not qualify. Consider the whole clip, including hazards later avoided. Do not assume a collision occurred or infer facts hidden outside the video.
  

質問

  At any point in this video, should the ego vehicle make an emergency stop to avoid a collision?
  

元の順番

  ["Emergency stop needed", "Emergency stop not needed"]
  

逆順

  ["Emergency stop not needed", "Emergency stop needed"]
  
モデル元の順番の回答逆順の回答想定回答との一致:元 / 逆
Jev-OmniEmergency stop not neededEmergency stop not needed除外 / 除外
ClefEmergency stop not neededEmergency stop not needed除外 / 除外
選択肢Jev-Omni:元Jev-Omni:逆Clef:元Clef:逆
Emergency stop needed0.0181250.2693260.1755380.310694
Emergency stop not needed0.9818750.7306740.8244620.689306
保存済み回答と実験ID
生成プロンプトと設定
  Four-second synthetic realistic forward-facing dashcam video, a fixed camera inside the ego vehicle, one continuous physically coherent shot, no cuts. Daylight on a closed road used only by empty fictional vehicles. No people, pedestrians, cyclists, visible occupants, injuries, blood, fire or graphic damage. No captions, text, labels, logos or watermarks. Show uninterrupted ego forward motion, visible road markings and changing vehicle separation before directly visible non-graphic vehicle-to-vehicle contact. An empty silver delivery van is visible at a right-side driveway from the first frame. Beginning at 0.4 seconds it reverses slowly out of the driveway, moves across the curb line, and stops with its rear half occupying the ego lane by 1.3 seconds. The ego vehicle continues advancing on the same straight route; the intrusion and rapidly shrinking gap remain visible before impact, providing a clear opportunity to brake hard to a stop. At about 3.0 seconds the ego front bumper visibly contacts the van rear-side corner and the van pivots slightly at the contact point. Keep coherent road geometry and directly visible vehicle boundaries. All evidence lies before 3.5 seconds.
  
  {
  "fixture_id": "C04",
  "cohorts": [
    "E4_NEAR_APPROACH_AMENDMENT"
  ],
  "modality": "video",
  "selected_attempt_id": "C04.attempt-1",
  "selected_attempt_index": 1,
  "generator_id": "minimax_h3_pdd8",
  "prompt_field": "prompt",
  "seed": 2026101404,
  "seed_status": "requested; no cross-hardware determinism claimed",
  "submitted_settings": {
    "seconds": 4,
    "aspect_ratio": "16:9",
    "seed": 2026101404
  },
  "configured_request_controls": {
    "num_inference_steps": 9,
    "flow_shift": 12.0,
    "audio_flow_shift": 3.0,
    "target": {
      "short_edge": 768,
      "aspect_ratio": "16:9"
    },
    "duration_seconds": 4,
    "fps": 24
  },
  "generation_request": {
    "path": "2026-10-02-clef-jev-collision/C04.attempt-1.generation.request.json",
    "sha256": "bc8932eb1f1d5fec94bc205a283f0a28aee0a2ad3401945d9ce03d57e547a75b",
    "bytes": 1311
  },
  "recorded_runtime_identity_source": {
    "path": "2026-10-02-clef-jev-collision/h3-loaded-identity.json",
    "sha256": "8682073cf08b2dde8ac45bd592c47a9a1aaede7c5e9e577804a7e6eb003f11be",
    "bytes": 4104
  },
  "native_asset": {
    "path": "2026-10-02-clef-jev-collision/attempts/C04.attempt-1/native.mp4",
    "sha256": "b7b373ade4f6b7bb6e396a3230b9315eea77ce1a72fa70051f035fcc0222df02",
    "bytes": 1942105
  },
  "evaluated_asset": {
    "path": "2026-10-02-clef-jev-near-approach/inputs/C04/asset.mp4",
    "sha256": "6f2d6fc6201a4866cdd10b2517e38c161382820b8214526f31fc26393c361959",
    "bytes": 2403091
  },
  "observed_media": {
    "native_frame_count": 107,
    "native_dimensions": null,
    "native_dimensions_notice": "Selected final inspection does not record native dimensions; configured target is 1344x768. No new probe was run.",
    "evaluated_transport_frames": 96,
    "evaluated_transport_fps": 24,
    "evaluated_transport_duration_seconds": 4
  },
  "inspection": {
    "path": "2026-10-02-clef-jev-near-approach/references/C04.attempt-1.final-inspected.json",
    "sha256": "58c5a633e04cfddd78f9ce593b4a2645d2b9ca40d51c2ec1ebb02cfb38724b5d",
    "bytes": 5392
  },
  "reference_classification": "ambiguous",
  "original_collision_media_accepted": false,
  "selection_notice": "Selected final attempt index 1 for each slot under a separately frozen near-approach amendment; no result-based selection. The original prompt requested contact, but visible contact was not independently confirmed."
}
  

C05:黄色いトラックへの接近

C05:評価に使用した先頭4秒、96フレーム、24 fps。 元のファイル

想定した答え:Emergency stop not needed

トラックとの車間は徐々に縮まりますが、間隔は残っています。急に危険が生じたり、接触が迫ったりする様子はなく、通常の減速で車間を調整する場面としました。これは、映っている4秒の範囲での判断です。

質問と選択肢

モデルに伝えた説明

  Judge only the visible driving sequence. An emergency stop means immediate, very hard braking intended to bring the ego vehicle to a stop to prevent an imminent collision; ordinary gradual braking or steering alone does not qualify. Consider the whole clip, including hazards later avoided. Do not assume a collision occurred or infer facts hidden outside the video.
  

質問

  At any point in this video, should the ego vehicle make an emergency stop to avoid a collision?
  

元の順番

  ["Emergency stop needed", "Emergency stop not needed"]
  

逆順

  ["Emergency stop not needed", "Emergency stop needed"]
  
モデル元の順番の回答逆順の回答想定回答との一致:元 / 逆
Jev-OmniEmergency stop not neededEmergency stop not needed一致 / 一致
ClefEmergency stop not neededEmergency stop not needed一致 / 一致
選択肢Jev-Omni:元Jev-Omni:逆Clef:元Clef:逆
Emergency stop needed0.0089850.1483550.0180550.035411
Emergency stop not needed0.9910150.8516450.9819450.964589
保存済み回答と実験ID
生成プロンプトと設定
  Four-second synthetic realistic forward-facing dashcam video, a fixed camera inside the ego vehicle, one continuous physically coherent shot, no cuts. Daylight on a closed road used only by empty fictional vehicles. No people, pedestrians, cyclists, visible occupants, injuries, blood, fire or graphic damage. No captions, text, labels, logos or watermarks. Show uninterrupted ego forward motion, visible road markings and changing vehicle separation before directly visible non-graphic vehicle-to-vehicle contact. On a slight uphill straight road an empty yellow box truck is centered in the ego lane directly ahead from the first frame. At 0.5 seconds the truck begins rolling backward toward the advancing ego vehicle. Show its rear wheels moving backward, its rear body enlarging rapidly, and a continuously shrinking open gap from 0.5 through 2.5 seconds. The truck remains entirely in the ego route with a sustained visible opportunity for immediate very hard braking to stop before the gap closes. At about 3.1 seconds the ego front bumper visibly contacts the truck rear bumper; the truck rocks slightly and relative motion abruptly couples at directly visible bumper contact. Keep all approach and contact evidence before 3.5 seconds.
  
  {
  "fixture_id": "C05",
  "cohorts": [
    "E4_NEAR_APPROACH_AMENDMENT"
  ],
  "modality": "video",
  "selected_attempt_id": "C05.attempt-1",
  "selected_attempt_index": 1,
  "generator_id": "minimax_h3_pdd8",
  "prompt_field": "prompt",
  "seed": 2026101405,
  "seed_status": "requested; no cross-hardware determinism claimed",
  "submitted_settings": {
    "seconds": 4,
    "aspect_ratio": "16:9",
    "seed": 2026101405
  },
  "configured_request_controls": {
    "num_inference_steps": 9,
    "flow_shift": 12.0,
    "audio_flow_shift": 3.0,
    "target": {
      "short_edge": 768,
      "aspect_ratio": "16:9"
    },
    "duration_seconds": 4,
    "fps": 24
  },
  "generation_request": {
    "path": "2026-10-02-clef-jev-collision/C05.attempt-1.generation.request.json",
    "sha256": "64b1d072d10a35f37340c4705bb19ef220d0e7460c309af52fb38d91df142840",
    "bytes": 1362
  },
  "recorded_runtime_identity_source": {
    "path": "2026-10-02-clef-jev-collision/h3-loaded-identity.json",
    "sha256": "8682073cf08b2dde8ac45bd592c47a9a1aaede7c5e9e577804a7e6eb003f11be",
    "bytes": 4104
  },
  "native_asset": {
    "path": "2026-10-02-clef-jev-collision/attempts/C05.attempt-1/native.mp4",
    "sha256": "55a1e1f01f4aa4ca5b1ccc631c036f9f50795a5726ad28cddfb44194961bbc79",
    "bytes": 2434210
  },
  "evaluated_asset": {
    "path": "2026-10-02-clef-jev-near-approach/inputs/C05/asset.mp4",
    "sha256": "fa9964489e1e78799a4db1e48a4e3cdab456e5add9fd9a3921e6c2a11b8e96c1",
    "bytes": 2953313
  },
  "observed_media": {
    "native_frame_count": 107,
    "native_dimensions": null,
    "native_dimensions_notice": "Selected final inspection does not record native dimensions; configured target is 1344x768. No new probe was run.",
    "evaluated_transport_frames": 96,
    "evaluated_transport_fps": 24,
    "evaluated_transport_duration_seconds": 4
  },
  "inspection": {
    "path": "2026-10-02-clef-jev-near-approach/references/C05.attempt-1.final-inspected.json",
    "sha256": "673cb28357ea8619e8f86bd8429eb39c041e0efbf47ec905eb56b5b73e14cda4",
    "bytes": 5392
  },
  "reference_classification": "not_needed",
  "original_collision_media_accepted": false,
  "selection_notice": "Selected final attempt index 1 for each slot under a separately frozen near-approach amendment; no result-based selection. The original prompt requested contact, but visible contact was not independently confirmed."
}
  

元の5本と合わせた結果

10本のうち、「緊急停止は不要」は4本、映像だけでは答えを決められないものは6本でした。「緊急停止が必要」と明確に判断できる動画は0本です。2つのモデルの回答は、元の20件と追加の20件を合わせて40件あります。

各モデルは集計対象の8回答すべてで一致し、残る12回答は除外しました。必要な場面の例がないため、必要なときに「必要」と答えられるかは、この結果では分かりません。記録のneeded_sensitivityもnullです。

データを作り直した記録

最初の動画生成は20回で、使えたのは5本でした。V03〜V07は用意できませんでした。条件を満たさなかった15回の生成結果は、モデルの回答の集計には含めていません。

埋まらなかった枠予定した場面
V03停止車列に近づき、対応が必要になる場面
V04対向車が自車の車線へ入る場面
V05ボールに続いて子どもが進路へ入る場面
V06柵によって道路から隔てられた子ども
V07自車が最初から停止したまま横断を待つ場面

コルク抜きI08は3回生成しても条件を満たさず、分類させる前にI08Rの卓上扇風機へ変更しました。I03のハンマーも作り直しています。A01の最初の試行には採用記録がなく、使ったのはattempt-01です。記録のない採用欄はnullで、生成の成否を表す値ではありません。

下の表は、最初の46試行と、接触動画の10試行の記録です。全項目は生成記録のJSONにあります。不採用になった動画のファイルは含まれていません。

最初の46試行
試行採用記録E1での使用要求したseed
A01-attempt-00記録なし不使用不明
A01-attempt-01採用使用不明
A02-attempt-00採用使用不明
A03-attempt-00採用使用不明
A04-attempt-00採用使用不明
A05-attempt-00採用使用不明
A06-attempt-00採用使用不明
A07-attempt-00採用使用不明
A08-attempt-00採用使用不明
A09-attempt-00採用使用不明
A10-attempt-00採用使用不明
I01-attempt-00採用使用不明
I02-attempt-00採用使用不明
I03-attempt-00不採用不使用2026100203
I03-attempt-01不採用不使用2026101203
I03-attempt-02採用使用不明
I04-attempt-00採用使用不明
I05-attempt-00採用使用不明
I06-attempt-00採用使用不明
I07-attempt-00採用使用不明
I08-attempt-00不採用不使用2026100208
I08-attempt-01不採用不使用2026101208
I08-attempt-02不採用不使用2026102208
I08R-attempt-00採用使用不明
I09-attempt-00採用使用不明
I10-attempt-00採用使用不明
V01-attempt-00採用使用不明
V02-attempt-00採用使用不明
V03-attempt-00不採用不使用2026100203
V03-attempt-01不採用不使用2026101203
V03-attempt-02不採用不使用2026102203
V04-attempt-00不採用不使用2026100204
V04-attempt-01不採用不使用2026101204
V04-attempt-02不採用不使用2026102204
V05-attempt-00不採用不使用2026100205
V05-attempt-01不採用不使用2026101205
V05-attempt-02不採用不使用2026102205
V06-attempt-00不採用不使用2026100206
V06-attempt-01不採用不使用2026101206
V07-attempt-00不採用不使用2026100207
V07-attempt-01不採用不使用2026101207
V07-attempt-02不採用不使用2026102207
V08-attempt-00採用使用不明
V09-attempt-00不採用不使用2026100209
V09-attempt-01採用使用不明
V10-attempt-00採用使用不明
試行接触動画としての採用E4の接近動画としての使用
C01.attempt-0不採用不使用
C01.attempt-1不採用使用
C02.attempt-0不採用不使用
C02.attempt-1不採用使用
C03.attempt-0不採用不使用
C03.attempt-1不採用使用
C04.attempt-0不採用不使用
C04.attempt-1不採用使用
C05.attempt-0不採用不使用
C05.attempt-1不採用使用
C01.attempt-0:不採用の理由と記録
  {
  "attempt_id": "C01.attempt-0",
  "fixture_id": "C01",
  "attempt_index": 0,
  "selected_for_E4": false,
  "generation_request": {
    "path": "2026-10-02-clef-jev-collision/C01.attempt-0.generation.request.json",
    "sha256": "683c37a6abd38a90d35fbbb56422cded06ab78548d0d47cdc26092db04c0fbf7",
    "bytes": 1223
  },
  "strict_contact_media_accepted": false,
  "strict_contact_score_eligible": false,
  "review": {
    "path": "2026-10-02-clef-jev-collision/attempts/C01.attempt-0/review.json",
    "sha256": "2bfabeeef18c76bc0724836694951ddfbc5c72b188c5ddd24c3693dcf25c2c6f",
    "bytes": 17089
  },
  "reason": "Rejected by the frozen media gate: the actual four-second transport shows a continuous approach to a red sedan with a decreasing apparent gap, but no unmistakable ego-to-vehicle contact. In full-resolution late frames the lower sedan/bumper becomes hidden behind the ego hood; its visible body remains aligned and intact, with no directly observable boundary contact, deformation, deflection or coherent collision consequence. Close following and hood occlusion alone cannot establish an impact. The native tail does not establish contact either and cannot rescue transport acceptance.",
  "reference_reason": "The visible closing conflict supports braking, especially in the late approach. However the actual clip does not establish contact, stopping/deceleration, calibrated ego speed or closing distance. It cannot reliably distinguish a necessary immediate very-hard stop from ordinary braking bringing the ego close behind the sedan. The reference therefore remains ambiguous; it does not assume a collision occurred, was unavoidable or would be prevented by a particular braking action."
}
  
C01.attempt-1:不採用の理由と記録
  {
  "attempt_id": "C01.attempt-1",
  "fixture_id": "C01",
  "attempt_index": 1,
  "selected_for_E4": true,
  "generation_request": {
    "path": "2026-10-02-clef-jev-collision/C01.attempt-1.generation.request.json",
    "sha256": "46f8aae33f19a8fcdc691864f7c247918a2f5d5d0ca502c65362e629c5d50cf4",
    "bytes": 1223
  },
  "strict_contact_media_accepted": false,
  "strict_contact_score_eligible": false,
  "review": {
    "path": "2026-10-02-clef-jev-collision/attempts/C01.attempt-1/review.json",
    "sha256": "276cff2dff398f71aa76290fdd6ee76378aeef04d8551529f91729f59f13f981",
    "bytes": 15780
  },
  "reason": "Rejected by the frozen media gate: the red sedan is approached continuously and its lower rear becomes hidden behind the ego hood near the end, but no unmistakable ego-to-vehicle contact appears in actual transport frames 0–95. The aligned rear body stays intact and its final position stabilizes without clear touch, deformation, deflection or coherent collision consequence. Hood occlusion does not establish impact.",
  "reference_reason": "Progressive closure and visible rear lamps support braking. Available stopping distance, calibrated closing speed and deceleration are unresolved; the sequence cannot reliably distinguish ordinary braking from a necessary immediate very-hard stop. Reference is ambiguous without assuming an impact from the close endpoint."
}
  
C02.attempt-0:不採用の理由と記録
  {
  "attempt_id": "C02.attempt-0",
  "fixture_id": "C02",
  "attempt_index": 0,
  "selected_for_E4": false,
  "generation_request": {
    "path": "2026-10-02-clef-jev-collision/C02.attempt-0.generation.request.json",
    "sha256": "eb4100e61789fd6b98bf7dbcd6b29b41df77c353c24834296b577391835af930",
    "bytes": 1258
  },
  "strict_contact_media_accepted": false,
  "strict_contact_score_eligible": false,
  "review": {
    "path": "2026-10-02-clef-jev-collision/attempts/C02.attempt-0/review.json",
    "sha256": "e06e7a592274a37288625de966bc96b13c6f84ca4044daca67b3b6b93222f430",
    "bytes": 16579
  },
  "reason": "Rejected by the frozen media gate: the white van grows larger as the apparent following gap decreases, but actual transport frames 0–95 do not show unmistakable ego-to-vehicle contact. Full-resolution frames 72, 84, 90 and 95 show an intact aligned rear; the lower van becomes hidden by the ego hood and the upper rear becomes cropped. No visible boundary contact, deformation, deflection or coherent collision consequence establishes impact.",
  "reference_reason": "The closing sequence supports braking, but the visible clip does not resolve ego speed, relative closing distance, available stopping distance or necessary braking intensity. Ordinary deceleration versus a necessary immediate very-hard stop cannot be distinguished reliably. Emergency-stop reference is ambiguous without assuming impact from hood occlusion."
}
  
C02.attempt-1:不採用の理由と記録
  {
  "attempt_id": "C02.attempt-1",
  "fixture_id": "C02",
  "attempt_index": 1,
  "selected_for_E4": true,
  "generation_request": {
    "path": "2026-10-02-clef-jev-collision/C02.attempt-1.generation.request.json",
    "sha256": "4d6e628c5ec435523792444799caee6c83eb25340a20a76a3cdf6aec3ba68618",
    "bytes": 1258
  },
  "strict_contact_media_accepted": false,
  "strict_contact_score_eligible": false,
  "review": {
    "path": "2026-10-02-clef-jev-collision/attempts/C02.attempt-1/review.json",
    "sha256": "b6b4899ea2a64cf42732b58e2dd53955eaffd68552aadf9504fb9796c67bae8b",
    "bytes": 15699
  },
  "reason": "Rejected by the frozen media gate: white van approaches continuously and fills the viewpoint, with its lower rear hidden by ego hood late in actual transport frames 0–95. No unmistakable direct contact, abrupt deflection, coherent impact deformation or collision consequence is visible. Rear panel shading does not establish an impact, and close hood occlusion alone is insufficient.",
  "reference_reason": "The van's shrinking apparent gap supports braking but does not resolve available stopping distance, closing speed or required intensity. The visible sequence cannot reliably distinguish ordinary deceleration from a necessary immediate very-hard stop. Reference remains ambiguous without treating the endpoint or panel shading as a collision."
}
  
C03.attempt-0:不採用の理由と記録
  {
  "attempt_id": "C03.attempt-0",
  "fixture_id": "C03",
  "attempt_index": 0,
  "selected_for_E4": false,
  "generation_request": {
    "path": "2026-10-02-clef-jev-collision/C03.attempt-0.generation.request.json",
    "sha256": "d2b2ae1ba3c77550ad11d3fff82d9a908668f4a6cc35788253968180f7f0bf1b",
    "bytes": 1320
  },
  "strict_contact_media_accepted": false,
  "strict_contact_score_eligible": false,
  "review": {
    "path": "2026-10-02-clef-jev-collision/attempts/C03.attempt-0/review.json",
    "sha256": "a1de42ec63c2da08961440fb4df4d0340ad05b6bd526355b6f8d0cb032236b11",
    "bytes": 16524
  },
  "reason": "Rejected by the frozen media gate: a blue pickup crosses from the left across the forward route and grows closer, but actual transport frames 0–95 show no unmistakable ego-to-vehicle contact. The body remains intact and the lower pickup disappears behind the hood. Close proximity and occlusion do not establish an impact; no clear boundary touch, deformation, deflection or coherent collision consequence is visible.",
  "reference_reason": "The visible crossing conflict calls for braking. The clip does not establish calibrated closing speed, stopping distance, deceleration or whether an immediate very-hard stop rather than ordinary braking was necessary and available. This reference is ambiguous; crossing and hood occlusion are not treated as proof of a collision or of a specific preventive action."
}
  
C03.attempt-1:不採用の理由と記録
  {
  "attempt_id": "C03.attempt-1",
  "fixture_id": "C03",
  "attempt_index": 1,
  "selected_for_E4": true,
  "generation_request": {
    "path": "2026-10-02-clef-jev-collision/C03.attempt-1.generation.request.json",
    "sha256": "5a99783a84d0c32faaee64a72c35383afda49e627e832bdd8e9a3aeb7995b415",
    "bytes": 1320
  },
  "strict_contact_media_accepted": false,
  "strict_contact_score_eligible": false,
  "review": {
    "path": "2026-10-02-clef-jev-collision/attempts/C03.attempt-1/review.json",
    "sha256": "17ba4958115c7c27bfde06a410e5ce4b424939e0d4c29cc793bd5aaca1f19fa3",
    "bytes": 16297
  },
  "reason": "The pickup is continuously approached while crossing the ego route, but its lower side and the possible contact zone become hidden behind the ego hood. A light crease or scuff on the front door is visible late; without visible vehicle boundaries meeting or coherent impact motion, that change does not establish unmistakable contact within frames 0–95. The frozen media gate therefore fails.",
  "reference_reason": "The crossing pickup creates a visible route conflict. The clip does not establish the speed, remaining stopping distance or urgency sufficiently to distinguish an immediate very hard stop from ordinary braking or steering. The possible late door crease cannot supply that prospective judgment; retain an ambiguous emergency-stop reference."
}
  
C04.attempt-0:不採用の理由と記録
  {
  "attempt_id": "C04.attempt-0",
  "fixture_id": "C04",
  "attempt_index": 0,
  "selected_for_E4": false,
  "generation_request": {
    "path": "2026-10-02-clef-jev-collision/C04.attempt-0.generation.request.json",
    "sha256": "94c2c3dbf7fd72978bd3047f01957176da48202be3ef8828376a70212bfd91da",
    "bytes": 1311
  },
  "strict_contact_media_accepted": false,
  "strict_contact_score_eligible": false,
  "review": {
    "path": "2026-10-02-clef-jev-collision/attempts/C04.attempt-0/review.json",
    "sha256": "b5df5ba40121a0ae518cb6a325634b49d9117b408f8fee7a2efa3c53911aff48",
    "bytes": 16590
  },
  "reason": "Rejected by the frozen media gate: a white van extends from a driveway on the right across the forward route and becomes close, but actual transport frames 0–95 do not show unmistakable vehicle contact. The lower van becomes hood-occluded; visible body panels remain intact and its late position stabilizes. No directly observable touch, deformation, deflection or coherent impact consequence is established.",
  "reference_reason": "The van obstructs the forward route and supports braking, but visible ego slowing and van motion do not resolve stopping distance or whether immediate very-hard stopping was required rather than ordinary braking. The close hood-occluded endpoint alone does not establish collision or the counterfactual efficacy of a stop. Reference remains ambiguous."
}
  
C04.attempt-1:不採用の理由と記録
  {
  "attempt_id": "C04.attempt-1",
  "fixture_id": "C04",
  "attempt_index": 1,
  "selected_for_E4": true,
  "generation_request": {
    "path": "2026-10-02-clef-jev-collision/C04.attempt-1.generation.request.json",
    "sha256": "bc8932eb1f1d5fec94bc205a283f0a28aee0a2ad3401945d9ce03d57e547a75b",
    "bytes": 1311
  },
  "strict_contact_media_accepted": false,
  "strict_contact_score_eligible": false,
  "review": {
    "path": "2026-10-02-clef-jev-collision/attempts/C04.attempt-1/review.json",
    "sha256": "1928136ebd43db05c67de369d3d3b00af14526e3b92cba2259d9ac8eaa0f025a",
    "bytes": 16100
  },
  "reason": "A white van extends from the right driveway into the ego route during a continuous approach. Through actual transport frame95, the visible van and ego hood do not establish vehicle boundaries meeting, deformation, deflection or abrupt coupled impact motion. Close approach is insufficient for the frozen collision media gate.",
  "reference_reason": "The emerging van creates a visible route conflict, but the clip does not establish relative speed, stopping distance or timing well enough to distinguish the need for immediate very hard stopping from ordinary braking or steering. Retain an ambiguous emergency-stop reference without inferring contact."
}
  
C05.attempt-0:不採用の理由と記録
  {
  "attempt_id": "C05.attempt-0",
  "fixture_id": "C05",
  "attempt_index": 0,
  "selected_for_E4": false,
  "generation_request": {
    "path": "2026-10-02-clef-jev-collision/C05.attempt-0.generation.request.json",
    "sha256": "f67e1d3d25b668bb44c26bf34d350099f2eb4bcdf459c1876f726a086de3203c",
    "bytes": 1362
  },
  "strict_contact_media_accepted": false,
  "strict_contact_score_eligible": false,
  "review": {
    "path": "2026-10-02-clef-jev-collision/attempts/C05.attempt-0/review.json",
    "sha256": "a741ecb2907b8c40740e66685fb354655b200181e1a7d472b5ff5da030fc85d7",
    "bytes": 16478
  },
  "reason": "Rejected by the frozen media gate: the yellow box truck grows larger ahead, but an unobscured strip of roadway remains between its intact rear bumper and the ego hood through actual transport frame 95. Full-resolution 72/84/90/95 show a clear gap, with no visible touch, deformation, deflection or coherent impact consequence.",
  "reference_reason": "The clip shows progressive closure to a truck without an abrupt visible obstruction or contact. Maintaining the gap may call for ordinary braking; calibrated speed, relative motion and stopping distance are not established well enough to determine necessary immediate very-hard stopping. Reference remains ambiguous instead of inferring urgency or rollback from the intended scenario."
}
  
C05.attempt-1:不採用の理由と記録
  {
  "attempt_id": "C05.attempt-1",
  "fixture_id": "C05",
  "attempt_index": 1,
  "selected_for_E4": true,
  "generation_request": {
    "path": "2026-10-02-clef-jev-collision/C05.attempt-1.generation.request.json",
    "sha256": "64b1d072d10a35f37340c4705bb19ef220d0e7460c309af52fb38d91df142840",
    "bytes": 1362
  },
  "strict_contact_media_accepted": false,
  "strict_contact_score_eligible": false,
  "review": {
    "path": "2026-10-02-clef-jev-collision/attempts/C05.attempt-1/review.json",
    "sha256": "747a0f0d7556565bc7813b32fe9f54a26931efdf4cf80b0e3066e0cf51050d97",
    "bytes": 16302
  },
  "reason": "The yellow box truck is approached gradually, but a substantial visible road gap remains between its rear bumper and the ego hood throughout actual transport frames0–95. No vehicle boundaries meet and no impact motion or consequence occurs. This candidate fails the frozen collision media gate.",
  "reference_reason": "The visible prefix shows gradual closure to a truck that remains ahead with a clear road gap, without sudden hazard onset, abrupt relative motion or a moment of visible imminent contact. Maintaining or restoring following distance through ordinary gradual braking is supported by this sequence; an immediate very hard stop is not visibly required. This qualitative not_needed annotation is confined to the visible prefix and is not inferred solely from the absence of collision. Rejected media remains ineligible for benchmark scoring."
}
  

実験データ

4つの実験で使った入力は40件、回答は160件です。Clefとの比較と、元の動画での緊急停止の実験は、最初の実験と同じ入力を使っています。

各回答のJSONには、選んだ答え、確率の全桁、実験ID、入力のハッシュがあります。

ファイル内容
E1_JEV_INITIAL.json35入力、70回答、未実行の動画5枠×2順序
E2_CLEF_COMMON.jsonClefの50回答と、E1の比較対象ID
E3_EMERGENCY_ORIGINAL.json元の動画5本への20回答と、集計からの除外
E4_NEAR_APPROACH_AMENDMENT.json追加動画5本への20回答と、集計からの除外
GENERATOR_PROVENANCE.json40入力のプロンプト、seed、設定、生成履歴
GENERATOR_PROVENANCE.md生成方法と記録の読み方
AUDIO_EXPERIMENT_MAPPING.md原稿、保存済みASR、音声の採用理由と不確定な部分
EXPORT_MANIFEST.json公開ファイルと元資料の対応、checksum、公開用の整理内容
VALIDATION.json元の公開セットの数値、hash、意味対応、除外条件の検証記録
SHA256SUMS.txt公開セット内の各ファイルのSHA-256
README_PUBLIC_EXPORT.md公開セットの範囲と解釈上の注意
元の構成案4種類の判定を中心にした英語の実験紹介
元の日本語動画原稿実験結果を紹介する動画用の原稿

音声はWAV、画像はJPEG、動画は評価に使った4秒のMP4です。生成元の長い動画やPNG、モデルの重み、実行ログは含まれていません。

今回の結果について

今回の結果は、実験用に作った少数のデータによるものです。使いたい用途で同じように答えられるかは、その用途に合うデータで確かめる必要があります。

参照先