单目3D初始代码

This commit is contained in:
zhao.zhu
2026-06-24 09:35:46 +08:00
commit 04a5895b6b
1153 changed files with 340700 additions and 0 deletions

3
tools/feishu_project/.gitignore vendored Executable file
View File

@@ -0,0 +1,3 @@
__pycache__/
*.pyc
reports/

160
tools/feishu_project/SKILL_fp.md Executable file
View File

@@ -0,0 +1,160 @@
---
name: feishu-project-issue-data
description: Use when working in the yolo26-3d repository and the user wants to read Feishu Project issue views, export a view to structured JSON, classify issue data addresses, download issue data, repair affected standard-path downloads, validate case completeness, or run batch inference on downloaded issue data. Covers fp CLI plus tools/feishu_project/export_feishu_view_issues.py, download_issue_data.py/sh, and run_issue_data_inference.py/sh, including the 董颖-G1Q3 workflow.
---
# Feishu Project Issue Data
Use the repo dev env for Python commands:
```bash
/root/.codex/skills/use-dongying-dev-env/scripts/with-dev-env.sh python ...
```
or:
```bash
/deeplearning_team/ydong/dongying/miniconda/envs/dev/bin/python ...
```
Current repo defaults are documented in [../../feishu_project.md](../../feishu_project.md).
## When to use
- Read or verify a Feishu Project issue view with `fp`
- Export a view such as `董颖-G1Q3` to structured JSON
- Work with `问题数据地址` / `问题数据地址_PDCL`
- Download issue data into the local workspace
- Repair historical bad copies that only copied `sigmastar.1`
- Check whether downloaded cases are inference-ready
- Batch-run exported-model inference on downloaded issue data
## Core workflow
### 1. Verify Feishu access and view contents
Use `fp` directly when the user wants current data.
```bash
fp view list -p <project_key> -u <user_key> -t issue --name "<view_name>"
fp workitem list -o json -p <project_key> -u <user_key> --view "<view_name>" --all
fp workitem get <issue_id> -o json -p <project_key> -u <user_key> -t issue
```
### 2. Export structured issue JSON
Use [../../export_feishu_view_issues.py](../../export_feishu_view_issues.py).
```bash
python ../../export_feishu_view_issues.py \
--project-key <project_key> \
--user-key <user_key> \
--view-name "<view_name>" \
--output ../../dongying_g1q3_issue_list.json
```
The export should include:
- `缺陷标签池`
- `问题数据地址`
- `问题数据地址_PDCL`
- `问题发生frameid`
### 3. Interpret data-address fields
Treat these as the same download class:
- pure `ADAS_...::...` clip references
- `mdi raw -r ...` commands
Use this rule:
- `ADAS_xxx::yyy` is equivalent to `mdi raw -r ADAS_xxx::yyy -s .`
Standard paths use these normalization rules:
- rewrite `hfs/project-G1M3` or `project-G1M3` to `G1M3` when needed
- if the path ends with `sigmastar.1`, copy the parent case dir
- if the path ends with `sigmastar.1/camera4.bin`, copy the case dir above it
- if the copied case has no local `test_data/calibs/camera4.json`, sync a shared parent `test_data` directory when present
### 4. Download or repair issue data
Use [../../download_issue_data.sh](../../download_issue_data.sh) or [../../download_issue_data.py](../../download_issue_data.py).
Common modes:
```bash
DRY_RUN=1 bash ../../download_issue_data.sh
```
```bash
ONLY_REDOWNLOAD_AFFECTED_CASES=1 bash ../../download_issue_data.sh
```
```bash
SKIP_MDI=1 bash ../../download_issue_data.sh --issue-id <id>
```
Defaults:
- download root: `/data1/dongying/Mono3d/G1Q3/feishu_project/downloaded_issue_data`
- manifest: `<download_root>/download_manifest.json`
Use `ONLY_REDOWNLOAD_AFFECTED_CASES=1` to repair old standard-path copies that previously kept only `sigmastar.1`.
### 5. Validate inference readiness
Use the same case-resolution rules as [../../../model_inference/adapters/video_dir_inference_utils.py](../../../model_inference/adapters/video_dir_inference_utils.py).
A valid case must resolve:
- `*/sigmastar.1/camera4.bin`
- a reachable `camera4.json` from one of:
- `case_dir/test_data/calibs/camera4.json`
- `case_dir.parent/test_data/calibs/camera4.json`
- `case_dir/sigmastar.1/calibs/camera4.json`
- `case_dir/calibs/camera4.json`
Prefer validating with the actual inference path-resolution logic instead of ad hoc file checks.
### 6. Run batch inference on downloaded issue data
Use [../../run_issue_data_inference.sh](../../run_issue_data_inference.sh) or [../../run_issue_data_inference.py](../../run_issue_data_inference.py).
```bash
DRY_RUN=1 bash ../../run_issue_data_inference.sh
```
```bash
bash ../../run_issue_data_inference.sh
```
Behavior:
- recursively scans the download root for `*/sigmastar.1/camera4.bin`
- calls [../../../model_inference/core/run_two_roi_exported_onnx_infer.py](../../../model_inference/core/run_two_roi_exported_onnx_infer.py) with `--video-case-dir`
- mirrors the download-tree relative layout into the inference output root
Defaults:
- inference root: `/data1/dongying/Mono3d/G1Q3/feishu_project/inference_issue_data`
- manifest: `<inference_root>/inference_manifest.json`
Useful flags:
- `SKIP_EXISTING=1`
- `ENABLE_ATTR=1`
- `SAVE_AGGREGATE_PREDICTIONS=1`
- `VIDEO_STRIDE=<n>`
- `MAX_IMAGES=<n>`
## Current repo artifacts
These files are useful outputs, but they are not the source of truth for latest Feishu data:
- [../../dongying_g1q3_issue_list.json](../../dongying_g1q3_issue_list.json)
- [../../dongying_g1q3_data_address_summary.md](../../dongying_g1q3_data_address_summary.md)
- [../../dongying_g1q3_data_address_catalog.md](../../dongying_g1q3_data_address_catalog.md)
If the user asks for latest status, re-query Feishu with `fp` and regenerate outputs instead of trusting stale local exports.

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,221 @@
#!/usr/bin/env python3
"""Build a synthetic issue JSON from a plain CNCAP case list."""
from __future__ import annotations
import argparse
import json
import re
import sys
from collections import Counter
from datetime import datetime
from pathlib import Path
from typing import Any
DEFAULT_INPUT = Path(__file__).with_name("cncap_case.txt")
DEFAULT_ID_BASE = 9_000_000_000
PDCL_REF_RE = re.compile(r"ADAS_[^:/\\\s]+::[^/\\\s]*")
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument(
"--input",
type=Path,
default=DEFAULT_INPUT,
help="Path to the CNCAP case list text file.",
)
parser.add_argument(
"--output",
type=Path,
required=True,
help="Path to the synthetic issue JSON output.",
)
parser.add_argument(
"--case-index-output",
type=Path,
default=None,
help="Optional companion index JSON output path.",
)
parser.add_argument(
"--id-base",
type=int,
default=DEFAULT_ID_BASE,
help="Synthetic issue id base. The first generated issue id is id_base + 1.",
)
return parser.parse_args()
def parse_case_line(raw_line: str, line_no: int) -> tuple[str, str] | None:
stripped = raw_line.strip()
if not stripped or stripped.startswith("#"):
return None
if "|" in stripped:
case_name, source = (part.strip() for part in stripped.split("|", 1))
else:
try:
case_name, source = stripped.rsplit(None, 1)
except ValueError as exc:
raise ValueError(
f"Line {line_no}: expected '<case_name> <case_ref>' or '<case_name> | <case_ref>', "
f"got: {raw_line.rstrip()!r}"
) from exc
if not case_name or not source:
raise ValueError(f"Line {line_no}: empty case name or case reference")
return case_name, source
def classify_source(source: str, line_no: int) -> tuple[str, str, str]:
if "mdi raw" in source.lower() or PDCL_REF_RE.search(source):
return "", source, "pdcl_mdi_download"
if "/" in source or "\\" in source:
return source, "", "standard_path"
raise ValueError(f"Line {line_no}: unsupported case reference format: {source!r}")
def build_payload(
*,
input_path: Path,
case_rows: list[dict[str, Any]],
id_base: int,
) -> tuple[dict[str, Any], dict[str, Any]]:
exported_at = datetime.now().astimezone().isoformat(timespec="seconds")
name_counter: Counter[str] = Counter()
items: list[dict[str, Any]] = []
index_cases: list[dict[str, Any]] = []
source_kind_counts: Counter[str] = Counter()
for ordinal, row in enumerate(case_rows, start=1):
case_name = row["case_name"]
source = row["source"]
line_no = row["line_no"]
raw_line = row["raw_line"]
name_counter[case_name] += 1
occurrence_index = name_counter[case_name]
issue_id = id_base + ordinal
standard_path, pdcl_ref, source_kind = classify_source(source, line_no)
source_kind_counts[source_kind] += 1
item = {
"id": issue_id,
"name": case_name,
"status": "CNCAP_CASE",
"created_at": None,
"updated_at": exported_at,
"问题数据地址": standard_path,
"问题数据地址_PDCL": pdcl_ref,
"问题发生frameid": None,
"source_line_no": line_no,
"case_ref": source,
"case_occurrence_index": occurrence_index,
"synthetic_issue_key": f"cncap_case_{ordinal:03d}",
}
items.append(item)
index_cases.append(
{
"id": issue_id,
"name": case_name,
"source_kind": source_kind,
"source_line_no": line_no,
"case_ref": source,
"case_occurrence_index": occurrence_index,
"raw_line": raw_line,
}
)
duplicate_names = sorted(name for name, count in name_counter.items() if count > 1)
payload = {
"exported_at": exported_at,
"source": {
"type": "cncap_case_txt",
"input_path": str(input_path.resolve()),
"id_base": id_base,
"work_item_type": "synthetic_issue",
"included_detail_fields": [
"问题数据地址",
"问题数据地址_PDCL",
"问题发生frameid",
],
},
"summary": {
"success": True,
"total": len(items),
"duplicate_name_count": len(duplicate_names),
"pdcl_case_count": source_kind_counts.get("pdcl_mdi_download", 0),
"standard_path_case_count": source_kind_counts.get("standard_path", 0),
},
"items": items,
}
case_index = {
"generated_at": exported_at,
"input_path": str(input_path.resolve()),
"id_base": id_base,
"total_cases": len(index_cases),
"duplicate_names": duplicate_names,
"cases": index_cases,
}
return payload, case_index
def write_json(path: Path, payload: dict[str, Any]) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(json.dumps(payload, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
def main() -> int:
args = parse_args()
input_path = args.input.resolve()
output_path = args.output.resolve()
case_index_output = (
args.case_index_output.resolve()
if args.case_index_output is not None
else output_path.with_name(f"{output_path.stem}.case_index.json")
)
if args.id_base < 0:
raise ValueError("--id-base must be greater than or equal to 0")
if not input_path.is_file():
raise FileNotFoundError(f"Input case list not found: {input_path}")
case_rows: list[dict[str, Any]] = []
for line_no, raw_line in enumerate(input_path.read_text(encoding="utf-8").splitlines(), start=1):
parsed = parse_case_line(raw_line, line_no)
if parsed is None:
continue
case_name, source = parsed
case_rows.append(
{
"line_no": line_no,
"raw_line": raw_line,
"case_name": case_name,
"source": source,
}
)
if not case_rows:
raise ValueError(f"No runnable cases were found in {input_path}")
payload, case_index = build_payload(
input_path=input_path,
case_rows=case_rows,
id_base=args.id_base,
)
write_json(output_path, payload)
write_json(case_index_output, case_index)
print(f"input: {input_path}")
print(f"output: {output_path}")
print(f"case_index_output: {case_index_output}")
print(f"total_cases: {payload['summary']['total']}")
print(f"duplicate_name_count: {payload['summary']['duplicate_name_count']}")
print(f"pdcl_case_count: {payload['summary']['pdcl_case_count']}")
print(f"standard_path_case_count: {payload['summary']['standard_path_case_count']}")
return 0
if __name__ == "__main__":
sys.exit(main())

View File

@@ -0,0 +1,326 @@
#!/usr/bin/env python3
"""Recover camera4.json for downloaded standard-path issue cases."""
from __future__ import annotations
import json
from dataclasses import dataclass
from pathlib import Path
from typing import Any, Iterable, Optional
TARGET_CALIB_REL = Path("test_data") / "calibs" / "camera4.json"
CALIB_ATTACHMENT_NAMES = (
"sigmastar.1/calibs/camera4.json",
"test_data/calibs/camera4.json",
"calibs/camera4.json",
)
CAMERA_CONFIG_FILE_NAME = "camera_config_folder.bin"
PREFERRED_CAMERA_IDS = (4, 0)
REQUIRED_FLAT_CALIB_KEYS = ("focal_u", "focal_v", "cu", "cv")
@dataclass(frozen=True)
class ExtractedCalibSource:
payload: dict[str, Any]
source_kind: str
source_path: Path
detail: str
@dataclass(frozen=True)
class CalibRecoveryResult:
status: str
detail: str
target_path: Path
source_path: Path | None = None
source_kind: str | None = None
def _safe_int(value: Any) -> int | None:
try:
if value is None:
return None
return int(value)
except (TypeError, ValueError):
return None
def _sorted_unique_paths(paths: Iterable[Path]) -> list[Path]:
unique = {path.resolve() if path.exists() else path for path in paths}
return sorted(unique, key=lambda path: (len(path.parts), str(path)))
def _camera_attachment_matches(name: str) -> bool:
normalized = str(name).strip().replace("\\", "/")
if normalized in CALIB_ATTACHMENT_NAMES:
return True
return normalized.endswith("/camera4.json") or normalized == "camera4.json"
def iter_concatenated_json_objects(text: str) -> list[dict[str, Any]]:
decoder = json.JSONDecoder()
pos = 0
payloads: list[dict[str, Any]] = []
text_len = len(text)
while pos < text_len:
while pos < text_len and (text[pos].isspace() or text[pos] == "\x00"):
pos += 1
if pos >= text_len:
break
obj, end = decoder.raw_decode(text, pos)
if isinstance(obj, dict):
payloads.append(obj)
pos = end
return payloads
def _coerce_float_list(value: Any, *, min_len: int = 0) -> list[float]:
if isinstance(value, dict):
ordered = [value.get("x"), value.get("y"), value.get("z")]
result = [float(item) for item in ordered if item is not None]
elif isinstance(value, (list, tuple)):
result = [float(item) for item in value]
else:
result = []
while len(result) < min_len:
result.append(0.0)
return result
def _is_valid_flat_calib_payload(payload: dict[str, Any]) -> bool:
return all(payload.get(key) is not None for key in REQUIRED_FLAT_CALIB_KEYS)
def _build_flat_camera4_payload(record: dict[str, Any], source_path: Path) -> dict[str, Any]:
payload = dict(record)
payload["focal_u"] = float(record["focal_u"])
payload["focal_v"] = float(record["focal_v"])
payload["cu"] = float(record["cu"])
payload["cv"] = float(record["cv"])
payload["roll"] = float(record.get("roll", 0.0))
payload["pitch"] = float(record.get("pitch", 0.0))
payload["yaw"] = float(record.get("yaw", 0.0))
payload["distort_coeffs"] = _coerce_float_list(record.get("distort_coeffs"))
payload["pos"] = _coerce_float_list(record.get("pos"), min_len=3)[:3]
if payload.get("image_width") is not None:
payload["image_width"] = int(payload["image_width"])
if payload.get("image_height") is not None:
payload["image_height"] = int(payload["image_height"])
payload.setdefault("camera_id", 4)
payload["recovered_from"] = CAMERA_CONFIG_FILE_NAME
payload["recovered_from_path"] = str(source_path)
return payload
def extract_calib_from_camera_config_folder(
config_path: Path,
preferred_camera_ids: tuple[int, ...] = PREFERRED_CAMERA_IDS,
) -> ExtractedCalibSource | None:
text = config_path.read_text(encoding="utf-8")
records = iter_concatenated_json_objects(text)
valid_records = [record for record in records if _is_valid_flat_calib_payload(record)]
if not valid_records:
return None
selected_records = valid_records
available_camera_ids = {
_safe_int(record.get("camera_id"))
for record in valid_records
if _safe_int(record.get("camera_id")) is not None
}
chosen_camera_id: int | None = None
for camera_id in preferred_camera_ids:
preferred_records = [
record for record in valid_records if _safe_int(record.get("camera_id")) == camera_id
]
if preferred_records:
selected_records = preferred_records
chosen_camera_id = camera_id
break
if chosen_camera_id is None:
if len(available_camera_ids) == 1:
chosen_camera_id = next(iter(available_camera_ids))
else:
counts: dict[int, int] = {}
for record in valid_records:
camera_id = _safe_int(record.get("camera_id"))
if camera_id is None:
continue
counts[camera_id] = counts.get(camera_id, 0) + 1
if counts:
chosen_camera_id = max(counts.items(), key=lambda item: item[1])[0]
selected_records = [
record for record in valid_records if _safe_int(record.get("camera_id")) == chosen_camera_id
]
best_record = max(
enumerate(selected_records),
key=lambda item: (_safe_int(item[1].get("utc_tick")) or -1, item[0]),
)[1]
payload = _build_flat_camera4_payload(best_record, config_path)
detail = (
f"camera_id={chosen_camera_id if chosen_camera_id is not None else 'unknown'} "
f"records={len(valid_records)} latest_utc_tick={best_record.get('utc_tick')}"
)
return ExtractedCalibSource(
payload=payload,
source_kind="camera_config_folder",
source_path=config_path,
detail=detail,
)
def _extract_calib_from_mcap_with_clip_reader(mcap_path: Path) -> ExtractedCalibSource | None:
try:
from pdcl_pyclip.reader import ClipReader
except ImportError:
return None
reader = ClipReader(str(mcap_path))
for attachment in reader.iter_attachments():
if _camera_attachment_matches(attachment.name):
payload = json.loads(attachment.data.decode("utf-8"))
return ExtractedCalibSource(
payload=payload,
source_kind="mcap_attachment",
source_path=mcap_path,
detail=f"attachment={attachment.name}",
)
return None
def _extract_calib_from_mcap_with_generic_reader(mcap_path: Path) -> ExtractedCalibSource | None:
try:
from mcap.reader import make_reader
except ImportError:
return None
with mcap_path.open("rb") as file:
reader = make_reader(file)
for attachment in reader.iter_attachments():
if _camera_attachment_matches(attachment.name):
payload = json.loads(attachment.data.decode("utf-8"))
return ExtractedCalibSource(
payload=payload,
source_kind="mcap_attachment",
source_path=mcap_path,
detail=f"attachment={attachment.name}",
)
return None
def extract_calib_from_mcap_attachment(mcap_path: Path) -> ExtractedCalibSource | None:
errors: list[str] = []
for extractor in (_extract_calib_from_mcap_with_clip_reader, _extract_calib_from_mcap_with_generic_reader):
try:
extracted = extractor(mcap_path)
except Exception as exc: # pragma: no cover - recovery fallback logging
errors.append(f"{extractor.__name__}: {type(exc).__name__}: {exc}")
continue
if extracted is not None:
return extracted
if errors:
raise RuntimeError("; ".join(errors))
return None
def _find_candidate_camera_config_paths(source_root: Path) -> list[Path]:
return _sorted_unique_paths(source_root.rglob(CAMERA_CONFIG_FILE_NAME))
def _find_candidate_mcap_paths(source_root: Path) -> list[Path]:
return _sorted_unique_paths(source_root.rglob("*.mcap"))
def _write_camera4_json(target_path: Path, payload: dict[str, Any]) -> None:
target_path.parent.mkdir(parents=True, exist_ok=True)
target_path.write_text(
json.dumps(payload, ensure_ascii=False, indent=2) + "\n",
encoding="utf-8",
)
def recover_camera4_json(
source_root: Path,
target_root: Path,
dry_run: bool = False,
) -> CalibRecoveryResult:
target_path = target_root / TARGET_CALIB_REL
if target_path.is_file():
return CalibRecoveryResult(
status="skipped_existing_calib",
detail=f"target already exists: {target_path}",
target_path=target_path,
)
camera_config_paths = _find_candidate_camera_config_paths(source_root)
mcap_paths = _find_candidate_mcap_paths(source_root)
errors: list[str] = []
for config_path in camera_config_paths:
try:
extracted = extract_calib_from_camera_config_folder(config_path)
except Exception as exc:
errors.append(f"{config_path}: {type(exc).__name__}: {exc}")
continue
if extracted is None:
continue
if dry_run:
return CalibRecoveryResult(
status="planned_calib_recovery_from_camera_config_folder",
detail=f"would write {target_path} from {config_path} ({extracted.detail})",
target_path=target_path,
source_path=config_path,
source_kind=extracted.source_kind,
)
_write_camera4_json(target_path, extracted.payload)
return CalibRecoveryResult(
status="recovered_calib_from_camera_config_folder",
detail=f"wrote {target_path} from {config_path} ({extracted.detail})",
target_path=target_path,
source_path=config_path,
source_kind=extracted.source_kind,
)
for mcap_path in mcap_paths:
try:
extracted = extract_calib_from_mcap_attachment(mcap_path)
except Exception as exc:
errors.append(f"{mcap_path}: {type(exc).__name__}: {exc}")
continue
if extracted is None:
continue
if dry_run:
return CalibRecoveryResult(
status="planned_calib_recovery_from_mcap_attachment",
detail=f"would write {target_path} from {mcap_path} ({extracted.detail})",
target_path=target_path,
source_path=mcap_path,
source_kind=extracted.source_kind,
)
_write_camera4_json(target_path, extracted.payload)
return CalibRecoveryResult(
status="recovered_calib_from_mcap_attachment",
detail=f"wrote {target_path} from {mcap_path} ({extracted.detail})",
target_path=target_path,
source_path=mcap_path,
source_kind=extracted.source_kind,
)
checked_sources = (
f"camera_config_folder.bin={len(camera_config_paths)} mcap={len(mcap_paths)}"
)
if errors:
checked_sources += f"; errors: {' | '.join(errors[:3])}"
return CalibRecoveryResult(
status="failed_calib_recovery",
detail=f"no recoverable calibration found under {source_root} ({checked_sources})",
target_path=target_path,
)

View File

@@ -0,0 +1,9 @@
CPLA-夜晚_AEB_40_5 ADAS_S5STNF0T406280R_20260409152111002995_edda23ef80bd::20260408001902
CSTA-LN_AEB_10_20 ADAS_S5STNF0T406280R_20260415153614565776_6304fc524d71::20260414173141
CSTA-LN_AEB_10_20 ADAS_S5STNF0T406280R_20260415153614565776_7b6bcdb90977::20260414172819
CSTA-LN_AEB_20_20 ADAS_S5STNF0T406280R_20260415153614565776_0ae4e25121ec::20260414171833
CSTA-LN_AEB_20_20 ADAS_S5STNF0T406280R_20260415153614565776_c66607e24083::20260414170603
CPLA-夜晚_AEB_FCW70_5 ADAS_S5STNF0T406280R_20260409152111002995_dd98b096d792::20260407233549
CPLA-夜晚_AEB_FCW60_5 ADAS_S5STNF0T406280R_20260409152111002995_8065a4b59db6::20260407233159
CBLA_AEB_FCW80_15 ADAS_S5STNF0T406280R_20260409152111002995_7254e457ef67::20260407115528
CBLA_AEB_FCW70_15 ADAS_S5STNF0T406280R_20260409152111002995_44c27781ddc3::20260407114731

View File

@@ -0,0 +1,838 @@
#!/usr/bin/env python3
"""Decode 61-frame windows around issue frame ids for D4Q2 network-share cases."""
from __future__ import annotations
import argparse
import io
import json
import re
import sys
from collections import Counter, deque
from dataclasses import dataclass
from datetime import datetime
from pathlib import Path
from typing import Any, Iterable, Optional
import cv2
import numpy as np
FILE = Path(__file__).resolve()
ROOT = FILE.parents[2]
if str(ROOT) not in sys.path:
sys.path.append(str(ROOT))
from tools.model_inference.adapters.video_dir_inference_utils import (
get_video_frame_info,
read_video_frame_index,
)
try:
from pdcl_pyclip.decoder_struct import StructDecoder
from pdcl_pyclip.msg_camera import VideoMessage
from pdcl_pyclip.reader import ClipReader
except ImportError:
ClipReader = None
StructDecoder = None
VideoMessage = None
NETWORK_SHARE_MARKERS = ("hfs.minieye.tech", "192.168.2.122")
WINDOW_RADIUS = 30
WINDOW_SIZE = WINDOW_RADIUS * 2 + 1
DEFAULT_INPUT_JSON = ROOT / "tools" / "feishu_project" / "dongying_d4q2_zhibao_issue_list.json"
DEFAULT_DOWNLOAD_ROOT = Path("/data1/dongying/Mono3d/D4Q2/feishu_project/downloaded_issue_data")
DEFAULT_OUTPUT_ROOT = Path("/data1/dongying/Mono3d/D4Q2/feishu_project/decoded_issue_frame_windows")
FRAME_ID_CAMERA4_RE = re.compile(r"camera4\s*:\s*(\d+)", re.IGNORECASE)
FRAME_ID_ANY_CAMERA_RE = re.compile(r"(camera\d+)\s*:\s*(\d+)", re.IGNORECASE)
PURE_DIGIT_RE = re.compile(r"^\d+$")
@dataclass(frozen=True)
class TargetFrame:
camera: str
frame_id: int
raw_text: str
@dataclass(frozen=True)
class CaseSource:
issue_id: int
issue_name: str
issue_dir: Path
path_dir: Path
case_dir: Path
relative_case_dir: Path
target_frame: TargetFrame
decode_all: bool
source_mode: str
source_paths: tuple[Path, ...]
@dataclass
class CaseResult:
issue_id: int
issue_name: str
case_dir: str
relative_case_dir: str
target_camera: str
target_frame_id: int | None
decode_mode: str
source_mode: str | None
source_paths: list[str]
output_dir: str
status: str
detail: str
matched_field: str | None = None
matched_frame_idx: int | None = None
matched_topic: str | None = None
extracted_count: int = 0
def to_dict(self) -> dict[str, Any]:
return {
"issue_id": self.issue_id,
"issue_name": self.issue_name,
"case_dir": self.case_dir,
"relative_case_dir": self.relative_case_dir,
"target_camera": self.target_camera,
"target_frame_id": self.target_frame_id,
"decode_mode": self.decode_mode,
"source_mode": self.source_mode,
"source_paths": self.source_paths,
"output_dir": self.output_dir,
"status": self.status,
"detail": self.detail,
"matched_field": self.matched_field,
"matched_frame_idx": self.matched_frame_idx,
"matched_topic": self.matched_topic,
"extracted_count": self.extracted_count,
}
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(
description="Decode 61-frame windows around issue frame ids for downloaded D4Q2 network-share cases."
)
parser.add_argument("--input-json", default=str(DEFAULT_INPUT_JSON))
parser.add_argument("--download-root", default=str(DEFAULT_DOWNLOAD_ROOT))
parser.add_argument("--output-root", default=str(DEFAULT_OUTPUT_ROOT))
parser.add_argument("--manifest-path", default="")
parser.add_argument("--issue-id", action="append", dest="issue_ids", type=int)
parser.add_argument("--decode-all-issue-id", action="append", dest="decode_all_issue_ids", type=int)
parser.add_argument("--window-radius", type=int, default=WINDOW_RADIUS)
parser.add_argument("--jpg-quality", type=int, default=95)
parser.add_argument("--dry-run", action="store_true")
parser.add_argument("--skip-existing", action="store_true")
return parser.parse_args()
def ensure_dir(path: Path, dry_run: bool) -> None:
if dry_run:
return
path.mkdir(parents=True, exist_ok=True)
def load_issue_items(path: Path) -> list[dict[str, Any]]:
payload = json.loads(path.read_text(encoding="utf-8"))
return payload["items"]
def parse_target_frame(frame_text: object) -> TargetFrame | None:
text = "" if frame_text is None else str(frame_text).strip()
if not text:
return None
camera4_match = FRAME_ID_CAMERA4_RE.search(text)
if camera4_match:
return TargetFrame(camera="camera4", frame_id=int(camera4_match.group(1)), raw_text=text)
if PURE_DIGIT_RE.fullmatch(text):
return TargetFrame(camera="any", frame_id=int(text), raw_text=text)
any_camera_match = FRAME_ID_ANY_CAMERA_RE.search(text)
if any_camera_match:
return TargetFrame(camera=any_camera_match.group(1).lower(), frame_id=int(any_camera_match.group(2)), raw_text=text)
return None
def is_network_share_issue(item: dict[str, Any]) -> bool:
address = str(item.get("问题数据地址") or "")
return any(marker in address for marker in NETWORK_SHARE_MARKERS)
def find_candidate_mcaps(case_dir: Path) -> list[Path]:
root_level = sorted(
path
for path in case_dir.iterdir()
if path.is_file() and path.suffix.lower() == ".mcap" and "_PB_" not in path.name
)
if root_level:
return root_level
recursive = sorted(
path
for path in case_dir.rglob("*.mcap")
if path.is_file() and "_PB_" not in path.name
)
return recursive
def find_camera4_bin(case_dir: Path) -> Path | None:
candidates = sorted(case_dir.rglob("camera4.bin"), key=lambda p: (len(p.relative_to(case_dir).parts), str(p)))
return candidates[0] if candidates else None
def discover_case_sources(
items: list[dict[str, Any]],
download_root: Path,
issue_filter: set[int] | None,
decode_all_issue_filter: set[int],
) -> tuple[list[CaseSource], list[CaseResult]]:
discovered: list[CaseSource] = []
skipped: list[CaseResult] = []
for item in items:
issue_id = int(item["id"])
if issue_filter and issue_id not in issue_filter:
continue
if not is_network_share_issue(item):
continue
issue_name = str(item["name"])
issue_dir = download_root / f"issue_{issue_id}"
target_frame = parse_target_frame(item.get("问题发生frameid"))
if target_frame is None:
skipped.append(
CaseResult(
issue_id=issue_id,
issue_name=issue_name,
case_dir="",
relative_case_dir="",
target_camera="",
target_frame_id=None,
decode_mode="window",
source_mode=None,
source_paths=[],
output_dir="",
status="skipped_missing_frame_id",
detail=f"unparseable frame id: {item.get('问题发生frameid')!r}",
)
)
continue
if target_frame.camera not in {"camera4", "camera1", "any"}:
skipped.append(
CaseResult(
issue_id=issue_id,
issue_name=issue_name,
case_dir="",
relative_case_dir="",
target_camera=target_frame.camera,
target_frame_id=target_frame.frame_id,
decode_mode="all" if issue_id in decode_all_issue_filter else "window",
source_mode=None,
source_paths=[],
output_dir="",
status="skipped_unsupported_camera",
detail=f"unsupported camera selector in frame id: {target_frame.raw_text}",
)
)
continue
if not issue_dir.is_dir():
skipped.append(
CaseResult(
issue_id=issue_id,
issue_name=issue_name,
case_dir=str(issue_dir),
relative_case_dir=str(issue_dir.name),
target_camera=target_frame.camera,
target_frame_id=target_frame.frame_id,
decode_mode="all" if issue_id in decode_all_issue_filter else "window",
source_mode=None,
source_paths=[],
output_dir="",
status="skipped_missing_issue_dir",
detail=f"issue download directory not found: {issue_dir}",
)
)
continue
path_dirs = sorted(path for path in issue_dir.iterdir() if path.is_dir() and path.name.startswith("path_"))
if not path_dirs:
skipped.append(
CaseResult(
issue_id=issue_id,
issue_name=issue_name,
case_dir=str(issue_dir),
relative_case_dir=str(issue_dir.name),
target_camera=target_frame.camera,
target_frame_id=target_frame.frame_id,
decode_mode="all" if issue_id in decode_all_issue_filter else "window",
source_mode=None,
source_paths=[],
output_dir="",
status="skipped_no_path_cases",
detail="no path_* directories found for network-share issue",
)
)
continue
for path_dir in path_dirs:
case_dirs = sorted(path for path in path_dir.iterdir() if path.is_dir())
if not case_dirs:
skipped.append(
CaseResult(
issue_id=issue_id,
issue_name=issue_name,
case_dir=str(path_dir),
relative_case_dir=str(path_dir.relative_to(download_root)),
target_camera=target_frame.camera,
target_frame_id=target_frame.frame_id,
decode_mode="all" if issue_id in decode_all_issue_filter else "window",
source_mode=None,
source_paths=[],
output_dir="",
status="skipped_empty_path_dir",
detail="path_* directory does not contain any case subdirectory",
)
)
continue
for case_dir in case_dirs:
relative_case_dir = case_dir.relative_to(download_root)
candidate_mcaps = find_candidate_mcaps(case_dir)
if candidate_mcaps:
discovered.append(
CaseSource(
issue_id=issue_id,
issue_name=issue_name,
issue_dir=issue_dir,
path_dir=path_dir,
case_dir=case_dir,
relative_case_dir=relative_case_dir,
target_frame=target_frame,
decode_all=issue_id in decode_all_issue_filter,
source_mode="mcap_stream",
source_paths=tuple(candidate_mcaps),
)
)
continue
camera4_bin = find_camera4_bin(case_dir)
if camera4_bin is not None:
discovered.append(
CaseSource(
issue_id=issue_id,
issue_name=issue_name,
issue_dir=issue_dir,
path_dir=path_dir,
case_dir=case_dir,
relative_case_dir=relative_case_dir,
target_frame=target_frame,
decode_all=issue_id in decode_all_issue_filter,
source_mode="camera4_bin",
source_paths=(camera4_bin,),
)
)
continue
skipped.append(
CaseResult(
issue_id=issue_id,
issue_name=issue_name,
case_dir=str(case_dir),
relative_case_dir=str(relative_case_dir),
target_camera=target_frame.camera,
target_frame_id=target_frame.frame_id,
decode_mode="all" if issue_id in decode_all_issue_filter else "window",
source_mode=None,
source_paths=[],
output_dir="",
status="skipped_no_source",
detail="no non-PB .mcap or camera4.bin found under case directory",
)
)
return discovered, skipped
def _plane_to_ndarray(plane) -> np.ndarray:
stride = plane.line_size
height = plane.height
width = plane.width
array = np.frombuffer(plane, dtype=np.uint8)
if stride == width:
return array.reshape(height, width)
return array.reshape(height, stride)[:, :width]
def _h265_payload_to_bgr(payload: bytes) -> np.ndarray:
try:
import av
except ImportError as exc:
raise ImportError("PyAV is required for MCAP frame decoding.") from exc
container = av.open(io.BytesIO(payload))
for frame in container.decode(video=0):
y_plane = _plane_to_ndarray(frame.planes[0])
u_plane = _plane_to_ndarray(frame.planes[1])
v_plane = _plane_to_ndarray(frame.planes[2])
uv_plane = np.zeros((u_plane.shape[0], u_plane.shape[1] * 2), dtype=np.uint8)
uv_plane[:, 0::2] = u_plane
uv_plane[:, 1::2] = v_plane
yuv_image = np.concatenate((y_plane.copy(), uv_plane), axis=0)
return cv2.cvtColor(yuv_image, cv2.COLOR_YUV2BGR_NV12)
raise ValueError("decode failed: no video frame in payload")
def iter_mcap_frames(mcap_paths: Iterable[Path], topic_candidates: tuple[str, ...]) -> Iterable[dict[str, Any]]:
if ClipReader is None or StructDecoder is None or VideoMessage is None:
raise ImportError("pdcl_pyclip is required for MCAP extraction.")
decoder = StructDecoder()
for mcap_path in mcap_paths:
reader = ClipReader(str(mcap_path))
for schema, channel, msg in reader.iter_messages():
data = decoder.decode(schema, channel, msg)
if not isinstance(data, VideoMessage):
continue
frame_id = getattr(data, "frame_id", None)
if frame_id is None:
continue
yield {
"source_path": mcap_path,
"topic": getattr(channel, "topic", ""),
"frame_id": int(frame_id),
"payload": data.payload,
"timestamp": str(getattr(data, "timestamp", "")),
}
def topic_matches_camera(topic: str, target_camera: str) -> bool:
normalized_topic = str(topic or "").lower()
if target_camera == "any":
return "camera" in normalized_topic
return target_camera in normalized_topic
def collect_mcap_window(
mcap_paths: tuple[Path, ...],
target_camera: str,
target_frame_id: int,
window_radius: int,
) -> tuple[list[dict[str, Any]], str, str]:
buffer_before: deque[dict[str, Any]] = deque(maxlen=window_radius)
selected: list[dict[str, Any]] = []
trailing_needed = window_radius
found = False
matched_topic = ""
for frame in iter_mcap_frames(mcap_paths, tuple()):
if not topic_matches_camera(frame["topic"], target_camera):
continue
if not found:
if frame["frame_id"] == target_frame_id:
selected = list(buffer_before) + [frame]
found = True
matched_topic = str(frame["topic"])
else:
buffer_before.append(frame)
else:
selected.append(frame)
trailing_needed -= 1
if trailing_needed <= 0:
break
if not found:
raise FileNotFoundError(f"target frame_id {target_frame_id} not found in MCAP sources for camera selector {target_camera}")
return selected, "frame_id", matched_topic
def collect_mcap_all_frames(
mcap_paths: tuple[Path, ...],
target_camera: str,
) -> tuple[list[dict[str, Any]], str]:
selected: list[dict[str, Any]] = []
matched_topic = ""
for frame in iter_mcap_frames(mcap_paths, tuple()):
if not topic_matches_camera(frame["topic"], target_camera):
continue
if not matched_topic:
matched_topic = str(frame["topic"])
selected.append(frame)
if not selected:
raise FileNotFoundError(f"no MCAP frames found for camera selector {target_camera}")
return selected, matched_topic
def find_video_frame_match(index_payload: Optional[dict[str, Any]], target_frame_id: int) -> tuple[int | None, str | None]:
if not index_payload:
return None, None
fields = index_payload.get("fields", {}) or {}
index_list = index_payload.get("index", []) or []
for field_name in ("frame_id", "cve_frame_id"):
if field_name not in fields:
continue
for frame_idx in range(len(index_list)):
info = get_video_frame_info(index_payload, frame_idx)
if info is None:
continue
value = info.get(field_name)
try:
if int(value) == target_frame_id:
return frame_idx, field_name
except (TypeError, ValueError):
continue
return None, None
def collect_video_window(
video_path: Path,
target_frame_id: int,
window_radius: int,
) -> tuple[list[dict[str, Any]], str, int]:
index_payload = read_video_frame_index(video_path)
matched_frame_idx, matched_field = find_video_frame_match(index_payload, target_frame_id)
if matched_frame_idx is None or matched_field is None:
raise FileNotFoundError(f"target frame_id {target_frame_id} not found in video index")
start_idx = max(0, matched_frame_idx - window_radius)
end_idx = matched_frame_idx + window_radius
cap = cv2.VideoCapture(str(video_path))
if not cap.isOpened():
raise RuntimeError(f"failed to open video file: {video_path}")
selected: list[dict[str, Any]] = []
try:
cap.set(cv2.CAP_PROP_POS_FRAMES, start_idx)
current_idx = start_idx
while current_idx <= end_idx:
ret, frame = cap.read()
if not ret:
break
frame_info = get_video_frame_info(index_payload, current_idx) if index_payload else None
selected.append(
{
"source_path": video_path,
"frame_idx": current_idx,
"frame_id": None if frame_info is None else frame_info.get("frame_id"),
"cve_frame_id": None if frame_info is None else frame_info.get("cve_frame_id"),
"timestamp": "" if frame_info is None else str(frame_info.get("timestamp", "")),
"image": frame,
}
)
current_idx += 1
finally:
cap.release()
return selected, matched_field, matched_frame_idx
def collect_video_all_frames(video_path: Path) -> tuple[list[dict[str, Any]], int]:
index_payload = read_video_frame_index(video_path)
cap = cv2.VideoCapture(str(video_path))
if not cap.isOpened():
raise RuntimeError(f"failed to open video file: {video_path}")
selected: list[dict[str, Any]] = []
current_idx = 0
try:
while True:
ret, frame = cap.read()
if not ret:
break
frame_info = get_video_frame_info(index_payload, current_idx) if index_payload else None
selected.append(
{
"source_path": video_path,
"frame_idx": current_idx,
"frame_id": None if frame_info is None else frame_info.get("frame_id"),
"cve_frame_id": None if frame_info is None else frame_info.get("cve_frame_id"),
"timestamp": "" if frame_info is None else str(frame_info.get("timestamp", "")),
"image": frame,
}
)
current_idx += 1
finally:
cap.release()
if not selected:
raise FileNotFoundError(f"no readable frames found in video file: {video_path}")
return selected, len(selected)
def save_decoded_frames(
issue_id: int,
output_dir: Path,
frames: list[dict[str, Any]],
target_frame_id: int,
window_radius: int,
jpg_quality: int,
dry_run: bool,
decode_all: bool,
) -> int:
images_dir = output_dir / ("frames_all" if decode_all else "frames_window")
if dry_run:
return len(frames)
ensure_dir(images_dir, dry_run=False)
encode_params = [int(cv2.IMWRITE_JPEG_QUALITY), int(jpg_quality)]
target_index = None if decode_all else min(window_radius, len(frames) - 1)
for index, frame in enumerate(frames):
offset = 0 if target_index is None else index - target_index
frame_id = frame.get("frame_id")
if frame_id is None:
frame_id = frame.get("cve_frame_id")
frame_id_token = "na" if frame_id is None else str(frame_id)
topic = str(frame.get("topic") or "")
camera_id_match = re.search(r"camera(\d+)", topic, re.IGNORECASE)
if camera_id_match:
camera_id_token = f"camera{camera_id_match.group(1)}"
else:
source_path = Path(str(frame.get("source_path") or ""))
camera_id_token = source_path.stem if source_path.stem else "camera"
if decode_all:
filename = f"{issue_id}_{camera_id_token}_{frame_id_token}.jpg"
else:
filename = f"{issue_id}_{camera_id_token}_{frame_id_token}.jpg"
image = frame.get("image")
if image is None:
image = _h265_payload_to_bgr(frame["payload"])
if not cv2.imwrite(str(images_dir / filename), image, encode_params):
raise IOError(f"failed to write image: {images_dir / filename}")
return len(frames)
def process_case(case: CaseSource, output_root: Path, window_radius: int, jpg_quality: int, dry_run: bool, skip_existing: bool) -> CaseResult:
output_dir = output_root / case.relative_case_dir
images_dir = output_dir / ("frames_all" if case.decode_all else "frames_window")
if skip_existing and images_dir.is_dir():
existing_images = [path for path in images_dir.iterdir() if path.is_file()]
if existing_images:
return CaseResult(
issue_id=case.issue_id,
issue_name=case.issue_name,
case_dir=str(case.case_dir),
relative_case_dir=str(case.relative_case_dir),
target_camera=case.target_frame.camera,
target_frame_id=case.target_frame.frame_id,
decode_mode="all" if case.decode_all else "window",
source_mode=case.source_mode,
source_paths=[str(path) for path in case.source_paths],
output_dir=str(output_dir),
status="skipped_existing",
detail="existing extracted frames found",
extracted_count=len(existing_images),
)
try:
if case.source_mode == "mcap_stream":
try:
if case.decode_all:
frames, matched_topic = collect_mcap_all_frames(case.source_paths, case.target_frame.camera)
matched_field = None
else:
frames, matched_field, matched_topic = collect_mcap_window(
case.source_paths,
case.target_frame.camera,
case.target_frame.frame_id,
window_radius,
)
extracted_count = save_decoded_frames(
case.issue_id, output_dir, frames, case.target_frame.frame_id, window_radius, jpg_quality, dry_run, case.decode_all
)
detail = (
f"decoded {extracted_count} frames from {len(case.source_paths)} mcap file(s)"
if case.decode_all
else f"decoded {extracted_count} frames from {len(case.source_paths)} mcap file(s)"
)
return CaseResult(
issue_id=case.issue_id,
issue_name=case.issue_name,
case_dir=str(case.case_dir),
relative_case_dir=str(case.relative_case_dir),
target_camera=case.target_frame.camera,
target_frame_id=case.target_frame.frame_id,
decode_mode="all" if case.decode_all else "window",
source_mode=case.source_mode,
source_paths=[str(path) for path in case.source_paths],
output_dir=str(output_dir),
status="planned" if dry_run else "decoded_mcap",
detail=("would " + detail) if dry_run else detail,
matched_field=matched_field,
matched_frame_idx=None,
matched_topic=matched_topic,
extracted_count=extracted_count,
)
except Exception as mcap_exc:
fallback_camera4 = find_camera4_bin(case.case_dir)
if fallback_camera4 is None or case.target_frame.camera not in {"camera4", "any"}:
raise mcap_exc
if case.decode_all:
frames, extracted_count_all = collect_video_all_frames(fallback_camera4)
matched_field = None
matched_frame_idx = None
extracted_count = save_decoded_frames(
case.issue_id, output_dir, frames, case.target_frame.frame_id, window_radius, jpg_quality, dry_run, case.decode_all
)
else:
frames, matched_field, matched_frame_idx = collect_video_window(
fallback_camera4,
case.target_frame.frame_id,
window_radius,
)
extracted_count = save_decoded_frames(
case.issue_id, output_dir, frames, case.target_frame.frame_id, window_radius, jpg_quality, dry_run, case.decode_all
)
detail = (
f"decoded {extracted_count} frames from camera4.bin fallback after mcap lookup failed: "
f"{type(mcap_exc).__name__}: {mcap_exc}"
)
return CaseResult(
issue_id=case.issue_id,
issue_name=case.issue_name,
case_dir=str(case.case_dir),
relative_case_dir=str(case.relative_case_dir),
target_camera=case.target_frame.camera,
target_frame_id=case.target_frame.frame_id,
decode_mode="all" if case.decode_all else "window",
source_mode="camera4_bin_fallback",
source_paths=[str(path) for path in case.source_paths] + [str(fallback_camera4)],
output_dir=str(output_dir),
status="planned" if dry_run else "decoded_camera4_bin_fallback",
detail=("would " + detail) if dry_run else detail,
matched_field=matched_field,
matched_frame_idx=matched_frame_idx,
matched_topic=None,
extracted_count=extracted_count,
)
if case.source_mode == "camera4_bin":
if case.decode_all:
frames, _ = collect_video_all_frames(case.source_paths[0])
matched_field, matched_frame_idx = None, None
else:
frames, matched_field, matched_frame_idx = collect_video_window(case.source_paths[0], case.target_frame.frame_id, window_radius)
extracted_count = save_decoded_frames(
case.issue_id, output_dir, frames, case.target_frame.frame_id, window_radius, jpg_quality, dry_run, case.decode_all
)
detail = f"decoded {extracted_count} frames from camera4.bin"
return CaseResult(
issue_id=case.issue_id,
issue_name=case.issue_name,
case_dir=str(case.case_dir),
relative_case_dir=str(case.relative_case_dir),
target_camera=case.target_frame.camera,
target_frame_id=case.target_frame.frame_id,
decode_mode="all" if case.decode_all else "window",
source_mode=case.source_mode,
source_paths=[str(path) for path in case.source_paths],
output_dir=str(output_dir),
status="planned" if dry_run else "decoded_camera4_bin",
detail=("would " + detail) if dry_run else detail,
matched_field=matched_field,
matched_frame_idx=matched_frame_idx,
matched_topic=None,
extracted_count=extracted_count,
)
return CaseResult(
issue_id=case.issue_id,
issue_name=case.issue_name,
case_dir=str(case.case_dir),
relative_case_dir=str(case.relative_case_dir),
target_camera=case.target_frame.camera,
target_frame_id=case.target_frame.frame_id,
decode_mode="all" if case.decode_all else "window",
source_mode=case.source_mode,
source_paths=[str(path) for path in case.source_paths],
output_dir=str(output_dir),
status="skipped_no_source",
detail=f"unsupported source mode: {case.source_mode}",
)
except Exception as exc:
return CaseResult(
issue_id=case.issue_id,
issue_name=case.issue_name,
case_dir=str(case.case_dir),
relative_case_dir=str(case.relative_case_dir),
target_camera=case.target_frame.camera,
target_frame_id=case.target_frame.frame_id,
decode_mode="all" if case.decode_all else "window",
source_mode=case.source_mode,
source_paths=[str(path) for path in case.source_paths],
output_dir=str(output_dir),
status="failed",
detail=f"{type(exc).__name__}: {exc}",
)
def build_manifest(args: argparse.Namespace, input_json: Path, download_root: Path, output_root: Path, discovered: list[CaseSource], results: list[CaseResult]) -> dict[str, Any]:
summary = Counter(result.status for result in results)
return {
"generated_at": datetime.now().astimezone().isoformat(timespec="seconds"),
"input_json": str(input_json),
"download_root": str(download_root),
"output_root": str(output_root),
"issue_filter": args.issue_ids or [],
"decode_all_issue_filter": args.decode_all_issue_ids or [],
"window_radius": args.window_radius,
"window_size": args.window_radius * 2 + 1,
"jpg_quality": args.jpg_quality,
"dry_run": args.dry_run,
"skip_existing": args.skip_existing,
"total_discovered_cases": len(discovered),
"summary": dict(summary),
"cases": [result.to_dict() for result in results],
}
def print_summary(manifest: dict[str, Any]) -> None:
print(f"input_json: {manifest['input_json']}")
print(f"download_root: {manifest['download_root']}")
print(f"output_root: {manifest['output_root']}")
print(f"window_size: {manifest['window_size']}")
print(f"dry_run: {manifest['dry_run']}")
print(f"total_discovered_cases: {manifest['total_discovered_cases']}")
for status, count in sorted(manifest["summary"].items()):
print(f"{status}: {count}")
def main() -> int:
args = parse_args()
input_json = Path(args.input_json).resolve()
download_root = Path(args.download_root).resolve()
output_root = Path(args.output_root).resolve()
manifest_path = (
Path(args.manifest_path).resolve()
if args.manifest_path
else output_root / "decode_manifest.json"
)
items = load_issue_items(input_json)
issue_filter = set(args.issue_ids) if args.issue_ids else None
decode_all_issue_filter = set(args.decode_all_issue_ids or [])
discovered, skipped = discover_case_sources(items, download_root, issue_filter, decode_all_issue_filter)
results = list(skipped)
for case in discovered:
results.append(process_case(case, output_root, args.window_radius, args.jpg_quality, args.dry_run, args.skip_existing))
manifest = build_manifest(args, input_json, download_root, output_root, discovered, results)
if not args.dry_run:
ensure_dir(manifest_path.parent, dry_run=False)
manifest_path.write_text(json.dumps(manifest, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
print_summary(manifest)
if args.dry_run:
print(f"manifest (not written in dry-run): {manifest_path}")
else:
print(f"manifest: {manifest_path}")
return 0
if __name__ == "__main__":
sys.exit(main())

View File

@@ -0,0 +1,45 @@
#!/usr/bin/env bash
set -euo pipefail
SCRIPT_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
PROJECT_ROOT=$(cd "${SCRIPT_DIR}/../.." && pwd)
INPUT_JSON=${INPUT_JSON:-"${PROJECT_ROOT}/tools/feishu_project/dongying_d4q2_zhibao_issue_list.json"}
DOWNLOAD_ROOT=${DOWNLOAD_ROOT:-/data1/dongying/Mono3d/D4Q2/feishu_project/downloaded_issue_data}
OUTPUT_ROOT=${OUTPUT_ROOT:-/data1/dongying/Mono3d/D4Q2/feishu_project/decoded_issue_frame_windows_v2}
MANIFEST_PATH=${MANIFEST_PATH:-"${OUTPUT_ROOT}/decode_manifest.json"}
PYTHON_BIN=${PYTHON_BIN:-/deeplearning_team/ydong/dongying/miniconda/envs/dev/bin/python}
WINDOW_RADIUS=${WINDOW_RADIUS:-30}
JPG_QUALITY=${JPG_QUALITY:-95}
DRY_RUN=${DRY_RUN:-0}
SKIP_EXISTING=${SKIP_EXISTING:-0}
DECODE_ALL_ISSUE_IDS=${DECODE_ALL_ISSUE_IDS:-}
CMD=(
"${PYTHON_BIN}" "${PROJECT_ROOT}/tools/feishu_project/decode_issue_frame_window.py"
--input-json "${INPUT_JSON}"
--download-root "${DOWNLOAD_ROOT}"
--output-root "${OUTPUT_ROOT}"
--manifest-path "${MANIFEST_PATH}"
--window-radius "${WINDOW_RADIUS}"
--jpg-quality "${JPG_QUALITY}"
)
if [[ "${DRY_RUN}" == "1" ]]; then
CMD+=(--dry-run)
fi
if [[ "${SKIP_EXISTING}" == "1" ]]; then
CMD+=(--skip-existing)
fi
if [[ -n "${DECODE_ALL_ISSUE_IDS}" ]]; then
# shellcheck disable=SC2206
ISSUE_ARR=(${DECODE_ALL_ISSUE_IDS})
for issue_id in "${ISSUE_ARR[@]}"; do
CMD+=(--decode-all-issue-id "${issue_id}")
done
fi
CMD+=("$@")
"${CMD[@]}"

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,805 @@
#!/usr/bin/env python3
"""Download issue data referenced by the Feishu issue export JSON."""
from __future__ import annotations
import argparse
import json
import re
import shutil
import subprocess
import sys
from collections import OrderedDict
from dataclasses import dataclass
from datetime import datetime
from pathlib import Path
from typing import Iterable
FILE = Path(__file__).resolve()
ROOT = FILE.parents[2]
if str(ROOT) not in sys.path:
sys.path.append(str(ROOT))
from tools.feishu_project.case_calib_recovery import recover_camera4_json
PDCL_REF_RE = re.compile(r"ADAS_[^:/\\\s]+::[^/\\\s]*")
MDI_RAW_REF_ARG_RE = re.compile(r"(?:^|\s)mdi\s+raw\b.*?(?:^|\s)-r\s+([^\s]+)")
STANDARD_PATH_SPLIT_RE = re.compile(r"[,\n;]+")
SHARED_CALIB_REL = Path("test_data") / "calibs" / "camera4.json"
PLACEHOLDER_TEXTS = {"待填", "待补充", "none", "null", "待提供"}
NETWORK_SHARE_PREFIX_MAPPINGS = (
("//hfs.minieye.tech/project-D4Q2", "/mnt/D4Q2"),
("//192.168.2.122/project-D4Q2", "/mnt/D4Q2"),
("//hfs.minieye.tech/project-G1M3", "/mnt/G1M3"),
("//192.168.2.122/project-G1M3", "/mnt/G1M3"),
("//hfs.minieye.tech/G1M3", "/mnt/G1M3"),
("//192.168.2.122/G1M3", "/mnt/G1M3"),
)
@dataclass
class ActionResult:
issue_id: int
issue_name: str
source_field: str
source_kind: str
raw_value: str | None
normalized_ref: str | None
output_dir: str
status: str
detail: str
resolved_source_path: str | None = None
command: list[str] | None = None
candidate_paths: list[str] | None = None
selected_subpath: str | None = None
def to_dict(self) -> dict:
return {
"issue_id": self.issue_id,
"issue_name": self.issue_name,
"source_field": self.source_field,
"source_kind": self.source_kind,
"raw_value": self.raw_value,
"normalized_ref": self.normalized_ref,
"output_dir": self.output_dir,
"status": self.status,
"detail": self.detail,
"resolved_source_path": self.resolved_source_path,
"command": self.command,
"candidate_paths": self.candidate_paths,
"selected_subpath": self.selected_subpath,
}
@dataclass(frozen=True)
class PDCLDownloadRequest:
normalized_ref: str
selected_subpath: str | None = None
raw_token: str | None = None
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(
description="Download or stage issue data from a Feishu issue export JSON."
)
parser.add_argument(
"--input-json",
default="tools/feishu_project/dongying_g1q3_issue_list.json",
help="Path to the issue export JSON.",
)
parser.add_argument(
"--output-root",
required=True,
help="Directory where downloaded or copied data should be stored.",
)
parser.add_argument(
"--manifest-path",
default=None,
help="Optional explicit path for the execution manifest JSON.",
)
parser.add_argument(
"--issue-id",
action="append",
dest="issue_ids",
type=int,
help="Optional issue id filter. Can be repeated.",
)
parser.add_argument(
"--dry-run",
action="store_true",
help="Plan actions without running mdi or copying files.",
)
parser.add_argument(
"--skip-mdi",
action="store_true",
help="Skip PDCL/MDI downloads and only process standard paths.",
)
parser.add_argument(
"--skip-copy",
action="store_true",
help="Skip standard-path copies and only process PDCL/MDI downloads.",
)
parser.add_argument(
"--only-redownload-affected-cases",
action="store_true",
help=(
"Only re-copy standard-path cases affected by the historical sigmastar.1/camera4.bin "
"copy bug. This mode skips PDCL/MDI downloads and replaces stale copied targets."
),
)
parser.add_argument(
"--skip-calib-recovery",
action="store_true",
help=(
"Skip recovering camera4.json from camera_config_folder.bin or mcap attachments "
"after standard-path copies."
),
)
return parser.parse_args()
def load_issue_items(path: Path) -> list[dict]:
payload = json.loads(path.read_text(encoding="utf-8"))
return payload["items"]
def ensure_dir(path: Path, dry_run: bool) -> None:
if dry_run:
return
path.mkdir(parents=True, exist_ok=True)
def log_progress(message: str) -> None:
timestamp = datetime.now().astimezone().strftime("%Y-%m-%d %H:%M:%S")
print(f"[download_issue_data {timestamp}] {message}", flush=True)
def compact_text(value: object, max_len: int = 96) -> str:
text = "" if value is None else str(value).strip()
text = re.sub(r"\s+", " ", text)
if len(text) <= max_len:
return text
return f"{text[: max_len - 3]}..."
def summarize_issue_results(results: list[ActionResult]) -> str:
if not results:
return "no actions"
summary: dict[str, int] = {}
for result in results:
summary[result.status] = summary.get(result.status, 0) + 1
return ", ".join(f"{status}={summary[status]}" for status in sorted(summary))
def normalize_issue_dirname(issue_id: int) -> str:
return f"issue_{issue_id}"
def iter_issue_fields(item: dict) -> Iterable[tuple[str, object]]:
yield "问题数据地址", item.get("问题数据地址")
yield "问题数据地址_PDCL", item.get("问题数据地址_PDCL")
def _normalize_pdcl_selected_subpath(raw_subpath: str) -> str | None:
cleaned = raw_subpath.strip().strip("/")
if not cleaned:
return None
candidate = Path(cleaned)
if candidate.is_absolute():
return None
if any(part in {"", ".", ".."} for part in candidate.parts):
return None
return str(candidate)
def _build_pdcl_request_from_token(token: str) -> PDCLDownloadRequest | None:
stripped = token.strip().strip("\"'`")
match = PDCL_REF_RE.match(stripped)
if match is None:
return None
normalized_ref = match.group(0)
suffix = stripped[match.end():]
selected_subpath = None
if suffix.startswith("/"):
raw_subpath = suffix[1:]
if raw_subpath and not raw_subpath.startswith("ADAS_"):
selected_subpath = _normalize_pdcl_selected_subpath(raw_subpath)
return PDCLDownloadRequest(
normalized_ref=normalized_ref,
selected_subpath=selected_subpath,
raw_token=stripped,
)
def extract_pdcl_requests(raw_value: object) -> list[PDCLDownloadRequest]:
if raw_value is None:
return []
text = str(raw_value).strip()
if not text:
return []
requests: list[PDCLDownloadRequest] = []
for segment in (part.strip() for part in STANDARD_PATH_SPLIT_RE.split(text)):
if not segment:
continue
mdi_match = MDI_RAW_REF_ARG_RE.search(segment)
if mdi_match is not None:
request = _build_pdcl_request_from_token(mdi_match.group(1))
if request is not None:
requests.append(request)
continue
search_pos = 0
while True:
match = PDCL_REF_RE.search(segment, search_pos)
if match is None:
break
normalized_ref = match.group(0)
suffix = segment[match.end():]
selected_subpath = None
if suffix.startswith("/") and not suffix[1:].startswith("ADAS_"):
selected_subpath = _normalize_pdcl_selected_subpath(suffix[1:])
requests.append(
PDCLDownloadRequest(
normalized_ref=normalized_ref,
selected_subpath=selected_subpath,
raw_token=segment,
)
)
break
requests.append(
PDCLDownloadRequest(
normalized_ref=normalized_ref,
selected_subpath=None,
raw_token=normalized_ref,
)
)
search_pos = match.end()
deduped: list[PDCLDownloadRequest] = []
seen = set()
for request in requests:
key = (request.normalized_ref, request.selected_subpath)
if key in seen:
continue
seen.add(key)
deduped.append(request)
return deduped
def extract_standard_paths(raw_value: object) -> list[str]:
if raw_value is None:
return []
text = str(raw_value).strip()
if not text:
return []
if text.lower() in PLACEHOLDER_TEXTS:
return []
if extract_pdcl_requests(text):
return []
if "/" not in text and "\\" not in text:
return []
parts = [part.strip() for part in STANDARD_PATH_SPLIT_RE.split(text)]
return [part for part in parts if part]
def normalize_standard_source_path(path: Path) -> Path:
normalized = path
if normalized.name == "camera4.bin" and normalized.parent.name == "sigmastar.1":
return normalized.parent.parent
if normalized.name == "sigmastar.1":
return normalized.parent
return normalized
def is_affected_standard_path(raw_path: str) -> bool:
raw_path_obj = Path(raw_path.strip())
return normalize_standard_source_path(raw_path_obj) != raw_path_obj
def normalize_share_path_separators(path_str: str) -> str:
normalized = path_str.strip().replace("\\", "/")
normalized = re.sub(r"/{3,}", "//", normalized)
return normalized
def rewrite_network_share_path(path_str: str) -> str | None:
normalized = normalize_share_path_separators(path_str)
for prefix_src, prefix_dst in NETWORK_SHARE_PREFIX_MAPPINGS:
if normalized.startswith(prefix_src):
return f"{prefix_dst}{normalized[len(prefix_src):]}"
return None
def build_path_candidates(raw_path: str) -> list[Path]:
candidates: list[str] = [raw_path]
normalized_share = normalize_share_path_separators(raw_path)
if normalized_share != raw_path:
candidates.append(normalized_share)
network_share_rewritten = rewrite_network_share_path(raw_path)
if network_share_rewritten:
candidates.append(network_share_rewritten)
for needle in ("hfs/project-G1M3", "project-G1M3"):
for candidate in list(candidates):
if needle in candidate:
candidates.append(candidate.replace(needle, "G1M3"))
normalized_candidates = [normalize_standard_source_path(Path(candidate)) for candidate in candidates]
unique_candidates = list(OrderedDict.fromkeys(str(candidate) for candidate in normalized_candidates))
return [Path(candidate) for candidate in unique_candidates]
def resolve_existing_path(raw_path: str) -> tuple[Path | None, list[Path]]:
candidates = build_path_candidates(raw_path)
for candidate in candidates:
if candidate.exists():
return candidate, candidates
return None, candidates
def remove_existing_target(path: Path) -> None:
if path.is_dir():
shutil.rmtree(path)
else:
path.unlink()
def copy_source_path(
source_path: Path,
output_dir: Path,
dry_run: bool,
replace_existing: bool = False,
legacy_target_names: Iterable[str] | None = None,
) -> tuple[str, str]:
target = output_dir / source_path.name
cleanup_targets: list[Path] = [target]
if legacy_target_names:
for target_name in legacy_target_names:
cleanup_targets.append(output_dir / target_name)
unique_cleanup_targets = list(OrderedDict.fromkeys(str(path) for path in cleanup_targets))
cleanup_paths = [Path(path) for path in unique_cleanup_targets]
existing_targets = [path for path in cleanup_paths if path.exists()]
if existing_targets and not replace_existing:
return "exists", f"target already exists: {target}"
if dry_run:
if existing_targets and replace_existing:
existing_str = ", ".join(str(path) for path in existing_targets)
return "planned_redownload", f"would replace {existing_str} with {source_path} -> {target}"
return "planned", f"would copy {source_path} -> {target}"
ensure_dir(output_dir, dry_run=False)
if replace_existing:
for existing_target in existing_targets:
remove_existing_target(existing_target)
if source_path.is_dir():
shutil.copytree(source_path, target)
else:
shutil.copy2(source_path, target)
if existing_targets and replace_existing:
replaced = ", ".join(str(path) for path in existing_targets)
return "redownloaded", f"replaced {replaced} with {target}"
return "copied", f"copied to {target}"
def build_copied_target_root(output_dir: Path, source_path: Path) -> Path:
return output_dir / source_path.name
def find_shared_test_data_dir(source_path: Path, max_parent_levels: int = 4) -> Path | None:
if (source_path / SHARED_CALIB_REL).is_file():
return None
current = source_path.parent
for _ in range(max_parent_levels):
candidate = current / "test_data"
if (candidate / "calibs" / "camera4.json").is_file():
return candidate
if current.parent == current:
break
current = current.parent
return None
def sync_shared_test_data(
source_path: Path,
target_root: Path,
dry_run: bool,
) -> tuple[str | None, str | None, str | None]:
shared_test_data_dir = find_shared_test_data_dir(source_path)
if shared_test_data_dir is None:
return None, None, None
target_shared_test_data_dir = target_root / "test_data"
target_shared_calib = target_shared_test_data_dir / "calibs" / "camera4.json"
if target_shared_calib.is_file():
return None, None, None
if dry_run:
return (
"planned_shared_calib_sync",
f"would copy shared test_data {shared_test_data_dir} -> {target_shared_test_data_dir}",
str(shared_test_data_dir),
)
shutil.copytree(shared_test_data_dir, target_shared_test_data_dir, dirs_exist_ok=True)
return (
"synced_shared_calib",
f"copied shared test_data {shared_test_data_dir} -> {target_shared_test_data_dir}",
str(shared_test_data_dir),
)
def recover_target_root_calib(
source_root: Path,
target_root: Path,
dry_run: bool,
) -> tuple[str, str, str | None]:
recovery = recover_camera4_json(
source_root=source_root,
target_root=target_root,
dry_run=dry_run,
)
return recovery.status, recovery.detail, None if recovery.source_path is None else str(recovery.source_path)
def _expected_pdcl_root_dir(output_dir: Path, ref: str) -> Path:
if "::" not in ref:
raise ValueError(f"Unexpected PDCL ref without '::': {ref}")
return output_dir / ref.split("::", 1)[1]
def _prune_pdcl_download_to_selected_subpath(
root_dir: Path,
selected_subpath: str,
dry_run: bool,
) -> tuple[str, str]:
selected_rel = Path(selected_subpath)
selected_path = root_dir / selected_rel
if dry_run:
return (
"planned_selected_subpath",
f"would keep {selected_path} and shared test_data under {root_dir}",
)
if not selected_path.exists():
return (
"failed_selected_subpath_missing",
f"selected subpath not found after mdi download: {selected_path}",
)
keep_names = {selected_rel.parts[0], "test_data"}
removed_children: list[str] = []
for child in root_dir.iterdir():
if child.name in keep_names:
continue
removed_children.append(child.name)
remove_existing_target(child)
detail = f"kept selected subpath {selected_path}"
if removed_children:
detail += f"; removed siblings: {', '.join(sorted(removed_children))}"
return "downloaded_selected_subpath", detail
def run_mdi_download(request: PDCLDownloadRequest, output_dir: Path, dry_run: bool) -> tuple[str, str, list[str]]:
command = ["mdi", "raw", "-r", request.normalized_ref, "-s", str(output_dir)]
if dry_run:
if request.selected_subpath:
root_dir = _expected_pdcl_root_dir(output_dir, request.normalized_ref)
status, detail = _prune_pdcl_download_to_selected_subpath(root_dir, request.selected_subpath, dry_run=True)
return status, f"would run {' '.join(command)}; {detail}", command
return "planned", f"would run {' '.join(command)}", command
ensure_dir(output_dir, dry_run=False)
completed = subprocess.run(
command,
check=False,
capture_output=True,
text=True,
encoding="utf-8",
)
if completed.returncode == 0:
base_detail = completed.stdout.strip() or "mdi raw completed"
if request.selected_subpath:
root_dir = _expected_pdcl_root_dir(output_dir, request.normalized_ref)
status, detail = _prune_pdcl_download_to_selected_subpath(root_dir, request.selected_subpath, dry_run=False)
return status, f"{base_detail}\n{detail}", command
return "downloaded", base_detail, command
detail = completed.stderr.strip() or completed.stdout.strip() or "mdi raw failed"
return "failed", detail, command
def process_issue(
item: dict,
output_root: Path,
dry_run: bool,
skip_mdi: bool,
skip_copy: bool,
only_redownload_affected_cases: bool,
skip_calib_recovery: bool,
) -> list[ActionResult]:
issue_id = int(item["id"])
issue_name = str(item["name"])
issue_dir = output_root / normalize_issue_dirname(issue_id)
results: list[ActionResult] = []
seen_pdcl_requests: set[tuple[str, str | None]] = set()
seen_paths: set[str] = set()
pdcl_index = 0
path_index = 0
for field_name, raw_value in iter_issue_fields(item):
if not skip_mdi and not only_redownload_affected_cases:
for request in extract_pdcl_requests(raw_value):
request_key = (request.normalized_ref, request.selected_subpath)
if request_key in seen_pdcl_requests:
results.append(
ActionResult(
issue_id=issue_id,
issue_name=issue_name,
source_field=field_name,
source_kind="pdcl_mdi_download",
raw_value=None if raw_value is None else str(raw_value),
normalized_ref=request.normalized_ref,
output_dir=str(issue_dir),
status="skipped_duplicate",
detail=(
f"duplicate PDCL ref: {request.normalized_ref}"
if request.selected_subpath is None
else f"duplicate PDCL ref+subpath: {request.normalized_ref} / {request.selected_subpath}"
),
selected_subpath=request.selected_subpath,
)
)
continue
seen_pdcl_requests.add(request_key)
pdcl_index += 1
download_dir = issue_dir / f"pdcl_{pdcl_index:02d}"
request_desc = request.normalized_ref
if request.selected_subpath:
request_desc += f"/{request.selected_subpath}"
log_progress(
f"issue_{issue_id} [download] pdcl_{pdcl_index:02d}: {compact_text(request_desc)}"
)
status, detail, command = run_mdi_download(request, download_dir, dry_run=dry_run)
results.append(
ActionResult(
issue_id=issue_id,
issue_name=issue_name,
source_field=field_name,
source_kind="pdcl_mdi_download",
raw_value=None if raw_value is None else str(raw_value),
normalized_ref=request.normalized_ref,
output_dir=str(download_dir),
status=status,
detail=detail,
command=command,
selected_subpath=request.selected_subpath,
)
)
if skip_copy:
continue
for raw_path in extract_standard_paths(raw_value):
if raw_path in seen_paths:
results.append(
ActionResult(
issue_id=issue_id,
issue_name=issue_name,
source_field=field_name,
source_kind="standard_path",
raw_value=raw_path,
normalized_ref=None,
output_dir=str(issue_dir),
status="skipped_duplicate",
detail=f"duplicate standard path: {raw_path}",
)
)
continue
seen_paths.add(raw_path)
path_index += 1
copy_dir = issue_dir / f"path_{path_index:02d}"
affected_standard_path = is_affected_standard_path(raw_path)
if only_redownload_affected_cases and not affected_standard_path:
continue
log_progress(
f"issue_{issue_id} [download] path_{path_index:02d}: {compact_text(raw_path)}"
)
resolved_source_path, candidates = resolve_existing_path(raw_path)
if resolved_source_path is None:
results.append(
ActionResult(
issue_id=issue_id,
issue_name=issue_name,
source_field=field_name,
source_kind="standard_path",
raw_value=raw_path,
normalized_ref=None,
output_dir=str(copy_dir),
status="skipped_missing",
detail="source path not found after rewrite attempts",
candidate_paths=[str(candidate) for candidate in candidates],
)
)
continue
legacy_target_names = []
if affected_standard_path:
legacy_target_names.append(Path(raw_path.strip()).name)
status, detail = copy_source_path(
resolved_source_path,
copy_dir,
dry_run=dry_run,
replace_existing=only_redownload_affected_cases and affected_standard_path,
legacy_target_names=legacy_target_names,
)
target_root = build_copied_target_root(copy_dir, resolved_source_path)
results.append(
ActionResult(
issue_id=issue_id,
issue_name=issue_name,
source_field=field_name,
source_kind="standard_path",
raw_value=raw_path,
normalized_ref=None,
output_dir=str(copy_dir),
status=status,
detail=detail,
resolved_source_path=str(resolved_source_path),
candidate_paths=[str(candidate) for candidate in candidates],
)
)
sync_status, sync_detail, shared_source_dir = sync_shared_test_data(
resolved_source_path,
target_root,
dry_run=dry_run,
)
if sync_status is not None:
results.append(
ActionResult(
issue_id=issue_id,
issue_name=issue_name,
source_field=field_name,
source_kind="shared_test_data",
raw_value=raw_path,
normalized_ref=None,
output_dir=str(target_root / "test_data"),
status=sync_status,
detail=sync_detail or "",
resolved_source_path=shared_source_dir,
candidate_paths=[str(candidate) for candidate in candidates],
)
)
if not skip_calib_recovery:
calib_status, calib_detail, calib_source_path = recover_target_root_calib(
source_root=resolved_source_path,
target_root=target_root,
dry_run=dry_run,
)
results.append(
ActionResult(
issue_id=issue_id,
issue_name=issue_name,
source_field=field_name,
source_kind="case_calib_recovery",
raw_value=raw_path,
normalized_ref=None,
output_dir=str(target_root / "test_data" / "calibs"),
status=calib_status,
detail=calib_detail,
resolved_source_path=calib_source_path,
candidate_paths=[str(candidate) for candidate in candidates],
)
)
return results
def build_manifest(
args: argparse.Namespace,
input_json: Path,
output_root: Path,
action_results: list[ActionResult],
) -> dict:
summary: dict[str, int] = {}
for action in action_results:
summary[action.status] = summary.get(action.status, 0) + 1
return {
"generated_at": datetime.now().astimezone().isoformat(timespec="seconds"),
"input_json": str(input_json),
"output_root": str(output_root),
"dry_run": args.dry_run,
"skip_mdi": args.skip_mdi,
"skip_copy": args.skip_copy,
"skip_calib_recovery": args.skip_calib_recovery,
"only_redownload_affected_cases": args.only_redownload_affected_cases,
"issue_filter": args.issue_ids or [],
"summary": summary,
"actions": [action.to_dict() for action in action_results],
}
def print_summary(manifest: dict) -> None:
print(f"input_json: {manifest['input_json']}")
print(f"output_root: {manifest['output_root']}")
print(f"dry_run: {manifest['dry_run']}")
for status, count in sorted(manifest["summary"].items()):
print(f"{status}: {count}")
def main() -> int:
args = parse_args()
input_json = Path(args.input_json).resolve()
output_root = Path(args.output_root).resolve()
manifest_path = (
Path(args.manifest_path).resolve()
if args.manifest_path
else output_root / "download_manifest.json"
)
items = load_issue_items(input_json)
if args.issue_ids:
issue_filter = set(args.issue_ids)
items = [item for item in items if int(item["id"]) in issue_filter]
ensure_dir(output_root, dry_run=args.dry_run)
action_results: list[ActionResult] = []
total_items = len(items)
log_progress(f"issues_to_process: {total_items}")
for index, item in enumerate(items, start=1):
issue_id = int(item["id"])
issue_name = compact_text(item.get("name"), max_len=64)
log_progress(f"[{index}/{total_items}] issue_{issue_id} start: {issue_name}")
issue_results = process_issue(
item=item,
output_root=output_root,
dry_run=args.dry_run,
skip_mdi=args.skip_mdi,
skip_copy=args.skip_copy,
only_redownload_affected_cases=args.only_redownload_affected_cases,
skip_calib_recovery=args.skip_calib_recovery,
)
action_results.extend(issue_results)
log_progress(
f"[{index}/{total_items}] issue_{issue_id} done: {summarize_issue_results(issue_results)}"
)
manifest = build_manifest(args, input_json, output_root, action_results)
ensure_dir(manifest_path.parent, dry_run=args.dry_run)
if not args.dry_run:
manifest_path.write_text(
json.dumps(manifest, ensure_ascii=False, indent=2) + "\n",
encoding="utf-8",
)
print_summary(manifest)
if args.dry_run:
print(f"manifest (not written in dry-run): {manifest_path}")
else:
print(f"manifest: {manifest_path}")
return 0
if __name__ == "__main__":
sys.exit(main())

View File

@@ -0,0 +1,38 @@
#!/usr/bin/env bash
set -euo pipefail
SCRIPT_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
PROJECT_ROOT=$(cd "${SCRIPT_DIR}/../.." && pwd)
SYNC_ROOT=${SYNC_ROOT:-/data1/dongying/Mono3d/G1Q3/feishu_project}
DEFAULT_INPUT_JSON="${SYNC_ROOT}/exports/dongying_g1q3_issue_list.json"
FALLBACK_INPUT_JSON="${PROJECT_ROOT}/tools/feishu_project/dongying_g1q3_issue_list.json"
INPUT_JSON=${INPUT_JSON:-"${DEFAULT_INPUT_JSON}"}
OUTPUT_ROOT=${OUTPUT_ROOT:-/data1/dongying/Mono3d/G1Q3/feishu_project/downloaded_issue_data}
MANIFEST_PATH=${MANIFEST_PATH:-"${OUTPUT_ROOT}/download_manifest.json"}
PYTHON_BIN=${PYTHON_BIN:-/deeplearning_team/ydong/dongying/miniconda/envs/dev/bin/python}
if [[ ! -f "${INPUT_JSON}" && -f "${FALLBACK_INPUT_JSON}" ]]; then
INPUT_JSON="${FALLBACK_INPUT_JSON}"
fi
EXTRA_ARGS=()
if [[ "${DRY_RUN:-0}" == "1" ]]; then
EXTRA_ARGS+=("--dry-run")
fi
if [[ "${SKIP_MDI:-0}" == "1" ]]; then
EXTRA_ARGS+=("--skip-mdi")
fi
if [[ "${SKIP_COPY:-0}" == "1" ]]; then
EXTRA_ARGS+=("--skip-copy")
fi
if [[ "${ONLY_REDOWNLOAD_AFFECTED_CASES:-0}" == "1" ]]; then
EXTRA_ARGS+=("--only-redownload-affected-cases")
fi
"${PYTHON_BIN}" "${PROJECT_ROOT}/tools/feishu_project/download_issue_data.py" \
--input-json "${INPUT_JSON}" \
--output-root "${OUTPUT_ROOT}" \
--manifest-path "${MANIFEST_PATH}" \
"${EXTRA_ARGS[@]}" \
"$@"

View File

@@ -0,0 +1,209 @@
#!/usr/bin/env python3
"""Export Feishu view issues and enrich them with selected detail fields."""
from __future__ import annotations
import argparse
import json
import subprocess
import sys
from datetime import datetime
from pathlib import Path
DEFAULT_EXTRA_FIELDS = [
"缺陷标签池",
"问题数据地址",
"问题数据地址_PDCL",
"问题发生frameid",
]
def run_fp(args: list[str]) -> dict:
result = subprocess.run(
["fp", *args],
check=False,
capture_output=True,
text=True,
encoding="utf-8",
)
if result.returncode != 0:
stderr = result.stderr.strip()
stdout = result.stdout.strip()
detail = stderr or stdout or "unknown error"
raise RuntimeError(f"fp command failed: {' '.join(args)}\n{detail}")
try:
return json.loads(result.stdout)
except json.JSONDecodeError as exc:
raise RuntimeError(
f"fp returned non-JSON output for command: {' '.join(args)}"
) from exc
def fetch_view(project_key: str, user_key: str, view_name: str, work_item_type: str) -> dict | None:
payload = run_fp(
[
"view",
"list",
"-o",
"json",
"-p",
project_key,
"-u",
user_key,
"-t",
work_item_type,
"--name",
view_name,
]
)
views = payload.get("data", [])
if not views:
return None
return views[0]
def fetch_items(project_key: str, user_key: str, view_name: str, work_item_type: str) -> dict:
return run_fp(
[
"workitem",
"list",
"-o",
"json",
"--all",
"-p",
project_key,
"-u",
user_key,
"-t",
work_item_type,
"--view",
view_name,
]
)
def fetch_item_detail(project_key: str, user_key: str, item_id: int, work_item_type: str) -> dict:
payload = run_fp(
[
"workitem",
"get",
str(item_id),
"-o",
"json",
"-p",
project_key,
"-u",
user_key,
"-t",
work_item_type,
]
)
return payload["data"]
def enrich_items(
base_items: list[dict],
project_key: str,
user_key: str,
work_item_type: str,
extra_fields: list[str],
) -> list[dict]:
enriched = []
for item in base_items:
detail = fetch_item_detail(project_key, user_key, item["id"], work_item_type)
detail_fields = detail.get("fields", {})
merged = dict(item)
for field_name in extra_fields:
merged[field_name] = detail_fields.get(field_name)
enriched.append(merged)
return enriched
def build_output(
project_key: str,
user_key: str,
view_name: str,
view: dict | None,
items_payload: dict,
extra_fields: list[str],
work_item_type: str,
) -> dict:
return {
"exported_at": datetime.now().astimezone().isoformat(timespec="seconds"),
"source": {
"project_key": project_key,
"user_key": user_key,
"view_name": view_name,
"view_id": None if view is None else view.get("view_id"),
"work_item_type": work_item_type,
"included_detail_fields": extra_fields,
},
"summary": {
"success": items_payload.get("success", False),
"page": items_payload.get("page"),
"total": items_payload.get("total"),
},
"items": items_payload["data"],
}
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--project-key", required=True)
parser.add_argument("--user-key", required=True)
parser.add_argument("--view-name", required=True)
parser.add_argument(
"--output",
required=True,
type=Path,
help="Path to the exported JSON file.",
)
parser.add_argument(
"--extra-field",
action="append",
dest="extra_fields",
help="Extra detail field to include. Can be repeated.",
)
parser.add_argument(
"--work-item-type",
default="issue",
help="Feishu work item type. Defaults to issue.",
)
return parser.parse_args()
def main() -> int:
args = parse_args()
extra_fields = args.extra_fields or list(DEFAULT_EXTRA_FIELDS)
view = fetch_view(args.project_key, args.user_key, args.view_name, args.work_item_type)
items_payload = fetch_items(args.project_key, args.user_key, args.view_name, args.work_item_type)
items_payload["data"] = enrich_items(
items_payload.get("data", []),
args.project_key,
args.user_key,
args.work_item_type,
extra_fields,
)
output = build_output(
args.project_key,
args.user_key,
args.view_name,
view,
items_payload,
extra_fields,
args.work_item_type,
)
args.output.parent.mkdir(parents=True, exist_ok=True)
args.output.write_text(
json.dumps(output, ensure_ascii=False, indent=2) + "\n",
encoding="utf-8",
)
return 0
if __name__ == "__main__":
sys.exit(main())

View File

@@ -0,0 +1,66 @@
`tools/feishu_project` 用来维护飞书问题视图导出、问题数据下载、批量推理,以及问题 case 的后处理脚本。
当前默认项目配置:
- `project_key`: `68ef617fb371dc80a10641f7`
- `user_key`: `7550145433285312514`
- `view_name`: `董颖-G1Q3`
推荐入口
- 全链路增量同步:`tools/feishu_project/sync_g1q3_issue_data.sh`
- CNCAP case 文本清单批处理:`tools/feishu_project/run_cncap_case_pipeline.sh`
- 只导出视图:`tools/feishu_project/export_feishu_view_issues.py`
- CNCAP case 清单转 synthetic issue JSON`tools/feishu_project/build_cncap_case_issue_json.py`
- 只下载数据:`tools/feishu_project/download_issue_data.sh`
- 只跑推理:`tools/feishu_project/run_issue_data_inference.sh`
- 问题标签画像统计:`tools/feishu_project/run_issue_tag_profile.sh`
- 发布问题标签画像内网页面:`tools/feishu_project/publish_issue_tag_profile_site.sh`
- 启动轻量静态服务:`tools/feishu_project/serve_issue_tag_profile_site.sh`
- 跟踪与后处理:`tools/feishu_project/run_issue_data_tracking.sh`
目录约定
- 此目录优先放源码和轻量文档,不再推荐把运行产物直接落在这里。
- 默认运行产物应写到 `/data1/dongying/Mono3d/*/feishu_project/` 下的 `exports/``downloaded_issue_data/``inference_issue_data/``reports/` 等目录。
- `__pycache__/`、HTML 报告、一次性导出 JSON/Markdown 结果属于运行产物或分析产物,除非明确要做样例留存,否则不要继续堆在 `tools/feishu_project/` 根目录。
维护原则
- 通用脚本不要保留单个 case、单个 issue 的硬编码默认值。
- G1Q3 包装脚本优先读取同步目录下的最新导出结果,不依赖仓库内静态 JSON。
- 复杂逻辑尽量收敛在 Python 主脚本里Shell 脚本只保留参数拼装和流程编排。
常用命令
```bash
# 查看 fp 帮助
fp --help
# 更新 fp
fp selfupdate
# 增量同步 G1Q3 问题并执行下载/推理/跟踪
bash tools/feishu_project/sync_g1q3_issue_data.sh
# 只下载指定 issue
bash tools/feishu_project/download_issue_data.sh --issue-id 6965833173
# 只对指定 issue 跑推理
bash tools/feishu_project/run_issue_data_inference.sh --issue-id 6965833173
# 基于最新 G1Q3 导出生成目标/问题标签画像
bash tools/feishu_project/run_issue_tag_profile.sh
# 生成标签画像并发布到静态站点目录
bash tools/feishu_project/publish_issue_tag_profile_site.sh
# 无 Nginx 时临时启动内网静态服务
bash tools/feishu_project/serve_issue_tag_profile_site.sh start
# 指定内网访问地址。该 IP 必须是当前机器网卡 IP或由反向代理/内网服务器承载。
INTRANET_HOST=192.168.2.169 bash tools/feishu_project/publish_issue_tag_profile_site.sh
INTRANET_HOST=192.168.2.169 bash tools/feishu_project/serve_issue_tag_profile_site.sh restart
```
内网页面发布
- 默认发布路径:`/data1/dongying/Mono3d/G1Q3/feishu_project/site/issue_tag_profile/index.html`
- 默认访问路径:`http://<内网机器IP>:8088/issue_tag_profile/`
- Nginx 示例配置:`tools/feishu_project/nginx_issue_tag_profile.conf.example`
- 临时静态服务脚本:`tools/feishu_project/serve_issue_tag_profile_site.sh`
- 可用 `INTRANET_HOST=<内网IP>``PUBLIC_HOST=<内网IP>` 指定页面里展示的访问地址;如果该 IP 不在当前机器网卡上,脚本会提示需要部署到对应内网机器或配置反向代理。

View File

@@ -0,0 +1,178 @@
#!/usr/bin/env python3
"""Recover missing camera4.json files for downloaded standard-path issue data."""
from __future__ import annotations
import argparse
import json
import sys
from datetime import datetime
from pathlib import Path
FILE = Path(__file__).resolve()
ROOT = FILE.parents[2]
if str(ROOT) not in sys.path:
sys.path.append(str(ROOT))
from tools.feishu_project.case_calib_recovery import CalibRecoveryResult, recover_camera4_json
DEFAULT_DOWNLOAD_ROOT = Path("/data1/dongying/Mono3d/G1Q3/feishu_project/downloaded_issue_data")
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument(
"--download-root",
default=str(DEFAULT_DOWNLOAD_ROOT),
help="Root directory containing downloaded issue data.",
)
parser.add_argument(
"--issue-id",
action="append",
dest="issue_ids",
type=int,
help="Only process the specified issue id. Can be repeated.",
)
parser.add_argument(
"--path-root",
action="append",
default=[],
help=(
"Optional explicit copied target root to recover. Can be repeated. "
"Example: downloaded_issue_data/issue_x/path_01/case_dir"
),
)
parser.add_argument(
"--manifest-path",
default="",
help="Optional explicit manifest path. Defaults to <download-root>/calib_recovery_manifest.json",
)
parser.add_argument("--dry-run", action="store_true")
return parser.parse_args()
def log_progress(message: str) -> None:
timestamp = datetime.now().astimezone().strftime("%Y-%m-%d %H:%M:%S")
print(f"[recover_downloaded_issue_calibs {timestamp}] {message}", flush=True)
def discover_target_roots(download_root: Path, issue_ids: list[int], explicit_roots: list[str]) -> list[Path]:
if explicit_roots:
return sorted({Path(path).resolve() for path in explicit_roots})
issue_dirs: list[Path]
if issue_ids:
issue_dirs = [download_root / f"issue_{issue_id}" for issue_id in sorted(set(issue_ids))]
else:
issue_dirs = sorted(path for path in download_root.glob("issue_*") if path.is_dir())
targets: list[Path] = []
seen: set[Path] = set()
for issue_dir in issue_dirs:
if not issue_dir.is_dir():
continue
for path_dir in sorted(path for path in issue_dir.glob("path_*") if path.is_dir()):
direct_case = path_dir / "sigmastar.1" / "camera4.bin"
if direct_case.is_file() and path_dir not in seen:
seen.add(path_dir)
targets.append(path_dir)
for child in sorted(path_dir.iterdir()):
if not child.is_dir() or child.name == "test_data":
continue
if child in seen:
continue
seen.add(child)
targets.append(child)
return targets
def result_to_dict(target_root: Path, result: CalibRecoveryResult) -> dict:
return {
"target_root": str(target_root),
"target_path": str(result.target_path),
"status": result.status,
"detail": result.detail,
"source_path": None if result.source_path is None else str(result.source_path),
"source_kind": result.source_kind,
}
def build_manifest(
download_root: Path,
dry_run: bool,
issue_ids: list[int],
target_roots: list[Path],
results: list[dict],
) -> dict:
summary: dict[str, int] = {}
for result in results:
status = result["status"]
summary[status] = summary.get(status, 0) + 1
return {
"generated_at": datetime.now().astimezone().isoformat(timespec="seconds"),
"download_root": str(download_root),
"dry_run": dry_run,
"issue_filter": issue_ids,
"target_roots": [str(path) for path in target_roots],
"summary": summary,
"results": results,
}
def print_summary(manifest: dict) -> None:
print(f"download_root: {manifest['download_root']}")
print(f"dry_run: {manifest['dry_run']}")
print(f"target_roots: {len(manifest['target_roots'])}")
for status, count in sorted(manifest["summary"].items()):
print(f"{status}: {count}")
def main() -> int:
args = parse_args()
download_root = Path(args.download_root).resolve()
manifest_path = (
Path(args.manifest_path).resolve()
if args.manifest_path
else download_root / "calib_recovery_manifest.json"
)
target_roots = discover_target_roots(download_root, args.issue_ids or [], args.path_root)
log_progress(f"targets_to_process: {len(target_roots)}")
results: list[dict] = []
for index, target_root in enumerate(target_roots, start=1):
log_progress(f"[{index}/{len(target_roots)}] start: {target_root}")
recovery = recover_camera4_json(
source_root=target_root,
target_root=target_root,
dry_run=args.dry_run,
)
results.append(result_to_dict(target_root, recovery))
log_progress(f"[{index}/{len(target_roots)}] done: {recovery.status}")
manifest = build_manifest(
download_root=download_root,
dry_run=args.dry_run,
issue_ids=args.issue_ids or [],
target_roots=target_roots,
results=results,
)
if not args.dry_run:
manifest_path.parent.mkdir(parents=True, exist_ok=True)
manifest_path.write_text(
json.dumps(manifest, ensure_ascii=False, indent=2) + "\n",
encoding="utf-8",
)
print_summary(manifest)
if args.dry_run:
print(f"manifest (not written in dry-run): {manifest_path}")
else:
print(f"manifest: {manifest_path}")
return 0
if __name__ == "__main__":
raise SystemExit(main())

View File

@@ -0,0 +1,330 @@
#!/usr/bin/env bash
set -euo pipefail
SCRIPT_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
PROJECT_ROOT=$(cd "${SCRIPT_DIR}/../.." && pwd)
CASE_LIST=${CASE_LIST:-"${PROJECT_ROOT}/tools/feishu_project/cncap_case.txt"}
SYNC_ROOT=${SYNC_ROOT:-/data1/dongying/Mono3d/CNCAP/feishu_project}
EXPORT_JSON=${EXPORT_JSON:-"${SYNC_ROOT}/exports/cncap_case_issue_list.json"}
CASE_INDEX_JSON=${CASE_INDEX_JSON:-"${SYNC_ROOT}/exports/cncap_case_issue_index.json"}
DOWNLOAD_ROOT=${DOWNLOAD_ROOT:-"${SYNC_ROOT}/downloaded_issue_data"}
DOWNLOAD_MANIFEST_PATH=${DOWNLOAD_MANIFEST_PATH:-"${DOWNLOAD_ROOT}/download_manifest.json"}
INFERENCE_ROOT=${INFERENCE_ROOT:-"${SYNC_ROOT}/inference_issue_data"}
INFERENCE_MANIFEST_PATH=${INFERENCE_MANIFEST_PATH:-"${INFERENCE_ROOT}/inference_manifest.json"}
PYTHON_BIN=${PYTHON_BIN:-/deeplearning_team/ydong/dongying/miniconda/envs/dev/bin/python}
RUN_BUILD=${RUN_BUILD:-1}
RUN_DOWNLOAD=${RUN_DOWNLOAD:-1}
RUN_INFERENCE=${RUN_INFERENCE:-1}
ENABLE_TRACKING=${ENABLE_TRACKING:-1}
ENABLE_CONVERT=${ENABLE_CONVERT:-1}
ENABLE_VISUALIZE=${ENABLE_VISUALIZE:-0}
ENABLE_TEMPORAL_ANALYSIS=${ENABLE_TEMPORAL_ANALYSIS:-0}
SKIP_EXISTING_INFERENCE=${SKIP_EXISTING_INFERENCE:-1}
DRY_RUN=${DRY_RUN:-0}
USE_ISSUE_FRAME_WINDOW=${USE_ISSUE_FRAME_WINDOW:-0}
FRAME_BEFORE=${FRAME_BEFORE:-200}
FRAME_AFTER=${FRAME_AFTER:-200}
MISSING_ISSUE_FRAME_POLICY=${MISSING_ISSUE_FRAME_POLICY:-full}
FRAME_INDEX_START=${FRAME_INDEX_START:-}
FRAME_INDEX_END=${FRAME_INDEX_END:-}
FRAME_ID_START=${FRAME_ID_START:-}
FRAME_ID_END=${FRAME_ID_END:-}
TARGET_FRAME_ID=${TARGET_FRAME_ID:-}
VIDEO_STRIDE=${VIDEO_STRIDE:-1}
MAX_IMAGES=${MAX_IMAGES:-0}
EXPORTED_MODEL=${EXPORTED_MODEL:-${PROJECT_ROOT}/runs/export/train_mono3d_two_roi_20260416-raw_no_edge/merged_model.torchscript}
DEVICE=${DEVICE:-}
PROVIDERS=${PROVIDERS:-}
ENABLE_ATTR=${ENABLE_ATTR:-0}
ENABLE_CROSS_CLASS_MERGE_PRIOR=${ENABLE_CROSS_CLASS_MERGE_PRIOR:-1}
SAVE_AGGREGATE_PREDICTIONS=${SAVE_AGGREGATE_PREDICTIONS:-1}
INFERENCE_EXTRA_ARGS=${INFERENCE_EXTRA_ARGS:-}
TRACK_CLASSES=${TRACK_CLASSES:-0 1 2 3 4 5 6 7 8 9 10 11 12}
IOU_THRESH=${IOU_THRESH:-0.3}
MAX_AGE=${MAX_AGE:-5}
MIN_HITS=${MIN_HITS:-1}
DIST_THRESH=${DIST_THRESH:-100}
MAX_3D_DISTANCE=${MAX_3D_DISTANCE:-10.0}
MAX_FRAMES=${MAX_FRAMES:-}
MERGE_OUTPUT_NAME=${MERGE_OUTPUT_NAME:-combined_tracking.json}
FILE_PATTERN=${FILE_PATTERN:-*.json}
ENABLE_USE_3D=${ENABLE_USE_3D:-0}
CONVERT_OUTPUT_DIR_NAME=${CONVERT_OUTPUT_DIR_NAME:-objectlist}
CONVERT_TRACKING_JSON_NAME=${CONVERT_TRACKING_JSON_NAME:-merge.json}
CONVERT_CAM_ID=${CONVERT_CAM_ID:-}
VIS_DOWNLOAD_ROOT=${VIS_DOWNLOAD_ROOT:-"${DOWNLOAD_ROOT}"}
VIS_OUTPUT_ROOT=${VIS_OUTPUT_ROOT:-}
VIS_OUTPUT_DIR_NAME=${VIS_OUTPUT_DIR_NAME:-tracking_vis_raw}
VIS_TRACKING_JSON_NAME=${VIS_TRACKING_JSON_NAME:-merge.json}
VIS_MAX_FRAMES=${VIS_MAX_FRAMES:-}
VIS_CLASS_ID=${VIS_CLASS_ID:-}
VIS_TRACK_IDS=${VIS_TRACK_IDS:-}
VIS_SHOW_TRAJECTORY=${VIS_SHOW_TRAJECTORY:-0}
TEMPORAL_OUTPUT_ROOT=${TEMPORAL_OUTPUT_ROOT:-}
TEMPORAL_OUTPUT_DIR_NAME=${TEMPORAL_OUTPUT_DIR_NAME:-temporal_observation}
TEMPORAL_JSON_NAME=${TEMPORAL_JSON_NAME:-merge.json}
TEMPORAL_MIN_LENGTH=${TEMPORAL_MIN_LENGTH:-3}
TEMPORAL_TRACK_IDS=${TEMPORAL_TRACK_IDS:-}
TEMPORAL_CLASS_ID=${TEMPORAL_CLASS_ID:-}
TEMPORAL_FRAME_ID_START=${TEMPORAL_FRAME_ID_START:-}
TEMPORAL_FRAME_ID_END=${TEMPORAL_FRAME_ID_END:-}
TEMPORAL_PLOTS=${TEMPORAL_PLOTS:-0}
TEMPORAL_EXPORT_SERIES=${TEMPORAL_EXPORT_SERIES:-1}
TEMPORAL_FOCUS_TRACK_PLOTS=${TEMPORAL_FOCUS_TRACK_PLOTS:-1}
TEMPORAL_PREFER_EGO=${TEMPORAL_PREFER_EGO:-1}
TEMPORAL_X_AXIS=${TEMPORAL_X_AXIS:-frame_id}
TEMPORAL_HEADING_SOURCE=${TEMPORAL_HEADING_SOURCE:-camera_reg}
TRACKING_MODEL_VERSION=${TRACKING_MODEL_VERSION:-}
SKIP_MDI=${SKIP_MDI:-0}
SKIP_COPY=${SKIP_COPY:-0}
ONLY_REDOWNLOAD_AFFECTED_CASES=${ONLY_REDOWNLOAD_AFFECTED_CASES:-0}
ID_BASE=${ID_BASE:-9000000000}
CASE_JSON_BUILDER=${CASE_JSON_BUILDER:-"${PROJECT_ROOT}/tools/feishu_project/build_cncap_case_issue_json.py"}
DOWNLOAD_WRAPPER=${DOWNLOAD_WRAPPER:-"${PROJECT_ROOT}/tools/feishu_project/download_issue_data.sh"}
INFERENCE_WRAPPER=${INFERENCE_WRAPPER:-"${PROJECT_ROOT}/tools/feishu_project/run_issue_data_inference.sh"}
TRACKING_WRAPPER=${TRACKING_WRAPPER:-"${PROJECT_ROOT}/tools/feishu_project/run_issue_data_tracking.sh"}
ISSUE_ID_ARGS=()
ISSUE_ID_VALUES=()
show_usage() {
cat <<EOF
Usage:
bash $(basename "${BASH_SOURCE[0]}") [--issue-id <id>]...
Environment variables:
CASE_LIST, SYNC_ROOT, EXPORT_JSON, CASE_INDEX_JSON
DOWNLOAD_ROOT, INFERENCE_ROOT, EXPORTED_MODEL
RUN_BUILD, RUN_DOWNLOAD, RUN_INFERENCE, ENABLE_TRACKING
ENABLE_CONVERT, ENABLE_VISUALIZE, ENABLE_TEMPORAL_ANALYSIS
DRY_RUN, ID_BASE
EOF
}
while (($# > 0)); do
case "$1" in
--issue-id)
if (($# < 2)); then
echo "Error: --issue-id requires a value" >&2
exit 1
fi
ISSUE_ID_ARGS+=("$1" "$2")
ISSUE_ID_VALUES+=("$2")
shift 2
;;
--issue-id=*)
issue_id="${1#*=}"
ISSUE_ID_ARGS+=("$1")
ISSUE_ID_VALUES+=("${issue_id}")
shift
;;
-h|--help)
show_usage
exit 0
;;
*)
echo "Error: unsupported option: $1" >&2
show_usage >&2
exit 1
;;
esac
done
print_stage() {
local stage_name="$1"
echo ""
echo "######################################################################"
echo "# ${stage_name}"
echo "######################################################################"
}
resolve_tracking_model_version() {
if [[ -n "${TRACKING_MODEL_VERSION}" ]]; then
printf '%s\n' "${TRACKING_MODEL_VERSION}"
return 0
fi
if [[ -n "${EXPORTED_MODEL}" ]] && [[ "${EXPORTED_MODEL}" =~ ([0-9]{8}) ]]; then
printf '%s\n' "${BASH_REMATCH[1]}"
return 0
fi
return 1
}
has_local_video_cases() {
local search_root="$1"
shift || true
if [[ ! -d "${search_root}" ]]; then
return 1
fi
if [[ "$#" -eq 0 ]]; then
[[ -n "$(find "${search_root}" -type f -path '*/sigmastar.1/camera4.bin' -print -quit)" ]]
return
fi
local issue_id=""
for issue_id in "$@"; do
if [[ -d "${search_root}/issue_${issue_id}" ]] && [[ -n "$(find "${search_root}/issue_${issue_id}" -type f -path '*/sigmastar.1/camera4.bin' -print -quit)" ]]; then
return 0
fi
done
return 1
}
TMP_ROOT=""
EFFECTIVE_EXPORT_JSON="${EXPORT_JSON}"
EFFECTIVE_CASE_INDEX_JSON="${CASE_INDEX_JSON}"
cleanup() {
if [[ -n "${TMP_ROOT}" && -d "${TMP_ROOT}" ]]; then
rm -rf "${TMP_ROOT}"
fi
}
trap cleanup EXIT
if [[ "${DRY_RUN}" == "1" && "${RUN_BUILD}" == "1" ]]; then
TMP_ROOT=$(mktemp -d /tmp/cncap_case_pipeline.XXXXXX)
EFFECTIVE_EXPORT_JSON="${TMP_ROOT}/cncap_case_issue_list.json"
EFFECTIVE_CASE_INDEX_JSON="${TMP_ROOT}/cncap_case_issue_index.json"
fi
export PYTHON_BIN
export TRACK_CLASSES
export IOU_THRESH
export MAX_AGE
export MIN_HITS
export DIST_THRESH
export MAX_3D_DISTANCE
export MAX_FRAMES
export MERGE_OUTPUT_NAME
export FILE_PATTERN
export ENABLE_USE_3D
export ENABLE_CONVERT
export ENABLE_VISUALIZE
export ENABLE_TEMPORAL_ANALYSIS
export CONVERT_OUTPUT_DIR_NAME
export CONVERT_TRACKING_JSON_NAME
export CONVERT_CAM_ID
export VIS_DOWNLOAD_ROOT
export VIS_OUTPUT_ROOT
export VIS_OUTPUT_DIR_NAME
export VIS_TRACKING_JSON_NAME
export VIS_MAX_FRAMES
export VIS_CLASS_ID
export VIS_TRACK_IDS
export VIS_SHOW_TRAJECTORY
export TEMPORAL_OUTPUT_ROOT
export TEMPORAL_OUTPUT_DIR_NAME
export TEMPORAL_JSON_NAME
export TEMPORAL_MIN_LENGTH
export TEMPORAL_TRACK_IDS
export TEMPORAL_CLASS_ID
export TEMPORAL_FRAME_ID_START
export TEMPORAL_FRAME_ID_END
export TEMPORAL_PLOTS
export TEMPORAL_EXPORT_SERIES
export TEMPORAL_FOCUS_TRACK_PLOTS
export TEMPORAL_PREFER_EGO
export TEMPORAL_X_AXIS
export TEMPORAL_HEADING_SOURCE
if [[ "${RUN_BUILD}" == "1" ]]; then
print_stage "Build CNCAP Synthetic Issue JSON"
echo "CASE_LIST : ${CASE_LIST}"
echo "EXPORT_JSON : ${EFFECTIVE_EXPORT_JSON}"
echo "CASE_INDEX_JSON : ${EFFECTIVE_CASE_INDEX_JSON}"
echo "ID_BASE : ${ID_BASE}"
"${PYTHON_BIN}" "${CASE_JSON_BUILDER}" \
--input "${CASE_LIST}" \
--output "${EFFECTIVE_EXPORT_JSON}" \
--case-index-output "${EFFECTIVE_CASE_INDEX_JSON}" \
--id-base "${ID_BASE}"
elif [[ ! -f "${EFFECTIVE_EXPORT_JSON}" ]]; then
echo "Error: export json does not exist and RUN_BUILD=0: ${EFFECTIVE_EXPORT_JSON}" >&2
exit 1
fi
if [[ "${RUN_DOWNLOAD}" == "1" ]]; then
print_stage "Download CNCAP Case Data"
INPUT_JSON="${EFFECTIVE_EXPORT_JSON}" \
OUTPUT_ROOT="${DOWNLOAD_ROOT}" \
MANIFEST_PATH="${DOWNLOAD_MANIFEST_PATH}" \
PYTHON_BIN="${PYTHON_BIN}" \
DRY_RUN="${DRY_RUN}" \
SKIP_MDI="${SKIP_MDI}" \
SKIP_COPY="${SKIP_COPY}" \
ONLY_REDOWNLOAD_AFFECTED_CASES="${ONLY_REDOWNLOAD_AFFECTED_CASES}" \
bash "${DOWNLOAD_WRAPPER}" "${ISSUE_ID_ARGS[@]}"
fi
if [[ "${RUN_INFERENCE}" == "1" ]]; then
if [[ "${DRY_RUN}" == "1" ]] && ! has_local_video_cases "${DOWNLOAD_ROOT}" "${ISSUE_ID_VALUES[@]}"; then
print_stage "Skip CNCAP Inference"
echo "Skipping inference in dry-run because no local downloaded camera4.bin cases were found."
echo "Run download without DRY_RUN first, or reuse an existing DOWNLOAD_ROOT to plan inference."
else
print_stage "Run CNCAP Inference"
DOWNLOAD_ROOT="${DOWNLOAD_ROOT}" \
OUTPUT_ROOT="${INFERENCE_ROOT}" \
MANIFEST_PATH="${INFERENCE_MANIFEST_PATH}" \
PYTHON_BIN="${PYTHON_BIN}" \
ISSUE_JSON="${EFFECTIVE_EXPORT_JSON}" \
VIDEO_STRIDE="${VIDEO_STRIDE}" \
MAX_IMAGES="${MAX_IMAGES}" \
FRAME_INDEX_START="${FRAME_INDEX_START}" \
FRAME_INDEX_END="${FRAME_INDEX_END}" \
FRAME_ID_START="${FRAME_ID_START}" \
FRAME_ID_END="${FRAME_ID_END}" \
TARGET_FRAME_ID="${TARGET_FRAME_ID}" \
FRAME_BEFORE="${FRAME_BEFORE}" \
FRAME_AFTER="${FRAME_AFTER}" \
USE_ISSUE_FRAME_WINDOW="${USE_ISSUE_FRAME_WINDOW}" \
MISSING_ISSUE_FRAME_POLICY="${MISSING_ISSUE_FRAME_POLICY}" \
EXPORTED_MODEL="${EXPORTED_MODEL}" \
DEVICE="${DEVICE}" \
PROVIDERS="${PROVIDERS}" \
ENABLE_ATTR="${ENABLE_ATTR}" \
ENABLE_CROSS_CLASS_MERGE_PRIOR="${ENABLE_CROSS_CLASS_MERGE_PRIOR}" \
SAVE_AGGREGATE_PREDICTIONS="${SAVE_AGGREGATE_PREDICTIONS}" \
SKIP_EXISTING="${SKIP_EXISTING_INFERENCE}" \
DRY_RUN="${DRY_RUN}" \
ENABLE_TRACKING=0 \
INFERENCE_EXTRA_ARGS="${INFERENCE_EXTRA_ARGS}" \
bash "${INFERENCE_WRAPPER}" "${ISSUE_ID_ARGS[@]}"
fi
fi
if [[ "${ENABLE_TRACKING}" != "1" ]]; then
exit 0
fi
if [[ "${DRY_RUN}" == "1" ]]; then
print_stage "Skip CNCAP Tracking"
echo "Skipping tracking because DRY_RUN=1."
exit 0
fi
TRACKING_MODEL_VERSION_RESOLVED=$(resolve_tracking_model_version || true)
print_stage "Run CNCAP Tracking/Convert Workflow"
echo "INFERENCE_ROOT : ${INFERENCE_ROOT}"
echo "VIS_DOWNLOAD_ROOT : ${VIS_DOWNLOAD_ROOT}"
if [[ -n "${TRACKING_MODEL_VERSION_RESOLVED}" ]]; then
echo "TRACKING_MODEL_VERSION : ${TRACKING_MODEL_VERSION_RESOLVED}"
fi
RESULTS_ROOT="${INFERENCE_ROOT}" \
PYTHON_BIN="${PYTHON_BIN}" \
TRACKING_MODEL_VERSION="${TRACKING_MODEL_VERSION_RESOLVED}" \
bash "${TRACKING_WRAPPER}" "${INFERENCE_ROOT}" "${ISSUE_ID_ARGS[@]}"

View File

@@ -0,0 +1,186 @@
#!/usr/bin/env bash
set -euo pipefail
SCRIPT_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
PROJECT_ROOT=$(cd "${SCRIPT_DIR}/../.." && pwd)
RESULTS_ROOT=${RESULTS_ROOT:-/data1/dongying/Mono3d/G1Q3/feishu_project/inference_issue_data}
OUTPUT_DIR_NAME=${OUTPUT_DIR_NAME:-objectlist}
PYTHON_BIN=${PYTHON_BIN:-/deeplearning_team/ydong/dongying/miniconda/envs/dev/bin/python}
CONVERT_LAUNCHER=${CONVERT_LAUNCHER:-"${PROJECT_ROOT}/tools/convert_merge_tracking_bundle/convert_merge_tracking_exported_onnx_infer_case.sh"}
MERGE_JSON_NAME=${MERGE_JSON_NAME:-merge.json}
CAM_ID=${CAM_ID:-}
TARGET_PATH=""
ISSUE_IDS=()
while (($# > 0)); do
case "$1" in
--issue-id)
if (($# < 2)); then
echo "Error: --issue-id requires a value" >&2
exit 1
fi
ISSUE_IDS+=("$2")
shift 2
;;
--issue-id=*)
ISSUE_IDS+=("${1#*=}")
shift
;;
-*)
echo "Error: unsupported option: $1" >&2
exit 1
;;
*)
if [[ -n "${TARGET_PATH}" ]]; then
echo "Error: multiple target paths provided: ${TARGET_PATH} and $1" >&2
exit 1
fi
TARGET_PATH="$1"
shift
;;
esac
done
if [[ -z "${TARGET_PATH}" ]]; then
TARGET_PATH="${RESULTS_ROOT}"
fi
if [[ ! -e "${TARGET_PATH}" ]]; then
echo "Error: target path does not exist: ${TARGET_PATH}" >&2
exit 1
fi
is_case_dir() {
local dir_path="$1"
[[ -d "${dir_path}" ]] && [[ -f "${dir_path}/${MERGE_JSON_NAME}" ]]
}
is_predictions_dir() {
local dir_path="$1"
[[ -d "${dir_path}" ]] || return 1
is_case_dir "$(dirname "${dir_path}")"
}
resolve_case_dir() {
local target_path="$1"
if is_case_dir "${target_path}"; then
printf '%s\n' "${target_path}"
return 0
fi
if is_predictions_dir "${target_path}"; then
dirname "${target_path}"
return 0
fi
if [[ -f "${target_path}" ]] && [[ "$(basename "${target_path}")" == "${MERGE_JSON_NAME}" ]]; then
dirname "${target_path}"
return 0
fi
return 1
}
case_output_dir() {
local case_dir="$1"
printf '%s/%s\n' "${case_dir%/}" "${OUTPUT_DIR_NAME}"
}
run_convert_case() {
local case_dir="$1"
local output_path
output_path="$(case_output_dir "${case_dir}")"
echo ""
echo "######################################################################"
echo "# Feishu issue-data protocol conversion"
echo "######################################################################"
echo "Case dir : ${case_dir}"
echo "Tracking JSON: ${case_dir%/}/${MERGE_JSON_NAME}"
echo "Output path : ${output_path}"
if [[ -n "${CAM_ID}" ]]; then
echo "Camera ID override: ${CAM_ID}"
fi
PYTHON_BIN="${PYTHON_BIN}" \
MERGE_JSON_NAME="${MERGE_JSON_NAME}" \
CAM_ID="${CAM_ID}" \
bash "${CONVERT_LAUNCHER}" "${case_dir}" "${output_path}"
}
run_convert_root() {
local target_root="$1"
local total_cases=0
local success_cases=0
local failed_cases=0
echo ""
echo "Batch-root mode: ${target_root}"
while IFS= read -r -d '' merge_json_path; do
local case_dir
case_dir="$(dirname "${merge_json_path}")"
((total_cases += 1))
if run_convert_case "${case_dir}"; then
((success_cases += 1))
else
((failed_cases += 1))
printf '[FAIL] case=%s\n' "${case_dir}" >&2
fi
done < <(
find "${target_root}" -type f -name "${MERGE_JSON_NAME}" -print0 | sort -z
)
if [[ "${total_cases}" -eq 0 ]]; then
echo "[ERROR] No ${MERGE_JSON_NAME} files were found under: ${target_root}" >&2
return 1
fi
echo ""
printf '[DONE] cases=%d success=%d failed=%d\n' \
"${total_cases}" "${success_cases}" "${failed_cases}"
[[ "${failed_cases}" -eq 0 ]]
}
if [[ "${#ISSUE_IDS[@]}" -eq 0 ]]; then
if RESOLVED_CASE_DIR="$(resolve_case_dir "${TARGET_PATH}")"; then
run_convert_case "${RESOLVED_CASE_DIR}"
else
run_convert_root "${TARGET_PATH}"
fi
exit 0
fi
total_issues=0
success_issues=0
failed_issues=0
for issue_id in "${ISSUE_IDS[@]}"; do
issue_root="${TARGET_PATH}/issue_${issue_id}"
((total_issues += 1))
if [[ ! -d "${issue_root}" ]]; then
echo "[FAIL] issue_${issue_id}: not found under ${TARGET_PATH}" >&2
((failed_issues += 1))
continue
fi
if run_convert_root "${issue_root}"; then
((success_issues += 1))
else
((failed_issues += 1))
echo "[FAIL] issue_${issue_id}: protocol conversion failed" >&2
fi
done
echo ""
printf '[DONE] issues=%d success=%d failed=%d\n' \
"${total_issues}" "${success_issues}" "${failed_issues}"
[[ "${failed_issues}" -eq 0 ]]

View File

@@ -0,0 +1,721 @@
#!/usr/bin/env python3
"""Batch inference runner for downloaded Feishu issue data."""
from __future__ import annotations
import argparse
import json
import re
import subprocess
import sys
from dataclasses import dataclass
from datetime import datetime
from pathlib import Path
from typing import Iterable
ROOT = Path(__file__).resolve().parents[2]
DEFAULT_DOWNLOAD_ROOT = Path("/data1/dongying/Mono3d/G1Q3/feishu_project/downloaded_issue_data")
DEFAULT_OUTPUT_ROOT = Path("/data1/dongying/Mono3d/G1Q3/feishu_project/inference_issue_data")
DEFAULT_INFERENCE_SCRIPT = ROOT / "tools" / "model_inference" / "core" / "run_two_roi_exported_onnx_infer.py"
DEFAULT_PYTHON_BIN = Path("/deeplearning_team/ydong/dongying/miniconda/envs/dev/bin/python")
ISSUE_DIR_RE = re.compile(r"^issue_(\d+)$")
SIGNED_INTEGER_RE = re.compile(r"^[+-]?\d+$")
FRAME_ID_CAMERA4_RE = re.compile(r"camera4\s*:\s*([+-]?\d+)", re.IGNORECASE)
FRAME_ID_ANY_CAMERA_RE = re.compile(r"(camera\d+)\s*:\s*([+-]?\d+)", re.IGNORECASE)
@dataclass(frozen=True)
class TargetFrame:
camera: str
frame_id: int
raw_text: str
@dataclass(frozen=True)
class InferenceCase:
issue_id: int
issue_dir: Path
case_dir: Path
camera4_bin: Path
relative_case_dir: Path
output_dir: Path
frame_window_source: str
target_frame_text: str | None = None
target_frame_id: int | None = None
requested_frame_index_start: int | None = None
requested_frame_index_end: int | None = None
requested_frame_id_start: int | None = None
requested_frame_id_end: int | None = None
@dataclass
class CaseResult:
issue_id: int
issue_dir: str
case_dir: str
camera4_bin: str
relative_case_dir: str
output_dir: str
status: str
detail: str
command: list[str]
log_path: str | None = None
frame_window_source: str | None = None
target_frame_text: str | None = None
target_frame_id: int | None = None
requested_frame_index_start: int | None = None
requested_frame_index_end: int | None = None
requested_frame_id_start: int | None = None
requested_frame_id_end: int | None = None
def to_dict(self) -> dict:
return {
"issue_id": self.issue_id,
"issue_dir": self.issue_dir,
"case_dir": self.case_dir,
"camera4_bin": self.camera4_bin,
"relative_case_dir": self.relative_case_dir,
"output_dir": self.output_dir,
"status": self.status,
"detail": self.detail,
"command": self.command,
"log_path": self.log_path,
"frame_window_source": self.frame_window_source,
"target_frame_text": self.target_frame_text,
"target_frame_id": self.target_frame_id,
"requested_frame_index_start": self.requested_frame_index_start,
"requested_frame_index_end": self.requested_frame_index_end,
"requested_frame_id_start": self.requested_frame_id_start,
"requested_frame_id_end": self.requested_frame_id_end,
}
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(
description="Run exported-model inference on all downloaded issue-data camera4.bin cases."
)
parser.add_argument(
"--download-root",
default=str(DEFAULT_DOWNLOAD_ROOT),
help="Root directory produced by download_issue_data.py.",
)
parser.add_argument(
"--output-root",
default=str(DEFAULT_OUTPUT_ROOT),
help="Root directory where inference outputs will be written.",
)
parser.add_argument(
"--manifest-path",
default="",
help="Optional explicit manifest JSON path. Defaults to <output-root>/inference_manifest.json",
)
parser.add_argument(
"--python-bin",
default=str(DEFAULT_PYTHON_BIN),
help="Python interpreter used to launch run_two_roi_exported_onnx_infer.py.",
)
parser.add_argument(
"--inference-script",
default=str(DEFAULT_INFERENCE_SCRIPT),
help="Path to run_two_roi_exported_onnx_infer.py.",
)
parser.add_argument(
"--issue-json",
default="",
help="Optional Feishu issue export JSON used to resolve 问题发生frameid windows.",
)
parser.add_argument(
"--issue-id",
action="append",
dest="issue_ids",
type=int,
help="Optional issue id filter. Can be repeated.",
)
parser.add_argument("--video-stride", type=int, default=1)
parser.add_argument("--max-images", type=int, default=0)
parser.add_argument("--frame-index-start", type=int, default=None)
parser.add_argument("--frame-index-end", type=int, default=None)
parser.add_argument("--frame-id-start", type=int, default=None)
parser.add_argument("--frame-id-end", type=int, default=None)
parser.add_argument("--target-frame-id", type=int, default=None)
parser.add_argument("--frame-before", type=int, default=100)
parser.add_argument("--frame-after", type=int, default=100)
parser.add_argument("--use-issue-frame-window", action="store_true")
parser.add_argument(
"--missing-issue-frame-policy",
choices=("full", "skip"),
default="full",
help="How to handle cases without a usable 问题发生frameid when --use-issue-frame-window is enabled.",
)
parser.add_argument("--exported-model", type=str, default="")
parser.add_argument("--device", type=str, default="")
parser.add_argument("--providers", nargs="*", default=None)
parser.add_argument("--enable-attr", action="store_true")
parser.add_argument("--enable-cross-class-merge-prior", action="store_true")
parser.add_argument("--save-aggregate-predictions", action="store_true")
parser.add_argument(
"--inference-arg",
action="append",
default=[],
help="Extra argument forwarded to run_two_roi_exported_onnx_infer.py. Can be repeated.",
)
parser.add_argument("--skip-existing", action="store_true")
parser.add_argument("--dry-run", action="store_true")
args = parser.parse_args()
if args.frame_before < 0:
parser.error("--frame-before must be greater than or equal to 0")
if args.frame_after < 0:
parser.error("--frame-after must be greater than or equal to 0")
if (
args.frame_index_start is not None
and args.frame_index_end is not None
and args.frame_index_start > args.frame_index_end
):
parser.error("--frame-index-start must be less than or equal to --frame-index-end")
if (
args.frame_id_start is not None
and args.frame_id_end is not None
and args.frame_id_start > args.frame_id_end
):
parser.error("--frame-id-start must be less than or equal to --frame-id-end")
if args.use_issue_frame_window and not args.issue_json:
parser.error("--issue-json is required when --use-issue-frame-window is enabled")
return args
def ensure_dir(path: Path, dry_run: bool) -> None:
if dry_run:
return
path.mkdir(parents=True, exist_ok=True)
def log_progress(message: str) -> None:
timestamp = datetime.now().astimezone().strftime("%Y-%m-%d %H:%M:%S")
print(f"[run_issue_data_inference {timestamp}] {message}", flush=True)
def compact_text(value: object, max_len: int = 96) -> str:
text = "" if value is None else str(value).strip()
text = re.sub(r"\s+", " ", text)
if len(text) <= max_len:
return text
return f"{text[: max_len - 3]}..."
def parse_issue_id_from_path(path: Path) -> int | None:
for part in path.parts:
match = ISSUE_DIR_RE.match(part)
if match:
return int(match.group(1))
return None
def load_issue_items(path: Path) -> list[dict]:
payload = json.loads(path.read_text(encoding="utf-8"))
return payload["items"]
def parse_target_frame(frame_text: object) -> tuple[TargetFrame | None, str]:
text = "" if frame_text is None else str(frame_text).strip()
if not text:
return None, "missing 问题发生frameid"
camera4_match = FRAME_ID_CAMERA4_RE.search(text)
if camera4_match:
frame_id = int(camera4_match.group(1))
if frame_id <= 0:
return None, f"non-positive 问题发生frameid: {text!r}"
return TargetFrame(camera="camera4", frame_id=frame_id, raw_text=text), ""
if SIGNED_INTEGER_RE.fullmatch(text):
frame_id = int(text)
if frame_id <= 0:
return None, f"non-positive 问题发生frameid: {text!r}"
return TargetFrame(camera="any", frame_id=frame_id, raw_text=text), ""
any_camera_match = FRAME_ID_ANY_CAMERA_RE.search(text)
if any_camera_match:
frame_id = int(any_camera_match.group(2))
if frame_id <= 0:
return None, f"non-positive 问题发生frameid: {text!r}"
return TargetFrame(camera=any_camera_match.group(1).lower(), frame_id=frame_id, raw_text=text), ""
return None, f"unparseable 问题发生frameid: {text!r}"
def build_issue_item_lookup(issue_json: Path) -> dict[int, dict]:
return {int(item["id"]): item for item in load_issue_items(issue_json)}
def has_manual_frame_window(args: argparse.Namespace) -> bool:
return any(
value is not None
for value in (
args.frame_index_start,
args.frame_index_end,
args.frame_id_start,
args.frame_id_end,
args.target_frame_id,
)
)
def build_case_result(
case: InferenceCase,
*,
status: str,
detail: str,
command: list[str] | None = None,
log_path: str | None = None,
) -> CaseResult:
return CaseResult(
issue_id=case.issue_id,
issue_dir=str(case.issue_dir),
case_dir=str(case.case_dir),
camera4_bin=str(case.camera4_bin),
relative_case_dir=str(case.relative_case_dir),
output_dir=str(case.output_dir),
status=status,
detail=detail,
command=command or [],
log_path=log_path,
frame_window_source=case.frame_window_source,
target_frame_text=case.target_frame_text,
target_frame_id=case.target_frame_id,
requested_frame_index_start=case.requested_frame_index_start,
requested_frame_index_end=case.requested_frame_index_end,
requested_frame_id_start=case.requested_frame_id_start,
requested_frame_id_end=case.requested_frame_id_end,
)
def discover_cases(download_root: Path, output_root: Path, issue_filter: set[int] | None) -> list[InferenceCase]:
cases: list[InferenceCase] = []
for camera4_bin in sorted(download_root.rglob("camera4.bin")):
if camera4_bin.parent.name != "sigmastar.1":
continue
try:
relative_camera = camera4_bin.relative_to(download_root)
except ValueError:
continue
issue_id = parse_issue_id_from_path(relative_camera)
if issue_id is None:
continue
if issue_filter and issue_id not in issue_filter:
continue
case_dir = camera4_bin.parent.parent
relative_case_dir = case_dir.relative_to(download_root)
issue_dir = download_root / f"issue_{issue_id}"
output_dir = output_root / relative_case_dir
cases.append(
InferenceCase(
issue_id=issue_id,
issue_dir=issue_dir,
case_dir=case_dir,
camera4_bin=camera4_bin,
relative_case_dir=relative_case_dir,
output_dir=output_dir,
frame_window_source="full",
)
)
return cases
def apply_frame_windows(
cases: list[InferenceCase],
args: argparse.Namespace,
issue_item_lookup: dict[int, dict],
) -> tuple[list[InferenceCase], list[CaseResult]]:
prepared_cases: list[InferenceCase] = []
skipped_results: list[CaseResult] = []
for case in cases:
prepared_case = InferenceCase(
issue_id=case.issue_id,
issue_dir=case.issue_dir,
case_dir=case.case_dir,
camera4_bin=case.camera4_bin,
relative_case_dir=case.relative_case_dir,
output_dir=case.output_dir,
frame_window_source="full",
)
if has_manual_frame_window(args):
requested_frame_id_start = args.frame_id_start
requested_frame_id_end = args.frame_id_end
if args.target_frame_id is not None:
if requested_frame_id_start is None:
requested_frame_id_start = max(0, args.target_frame_id - args.frame_before)
if requested_frame_id_end is None:
requested_frame_id_end = args.target_frame_id + args.frame_after
prepared_case = InferenceCase(
issue_id=case.issue_id,
issue_dir=case.issue_dir,
case_dir=case.case_dir,
camera4_bin=case.camera4_bin,
relative_case_dir=case.relative_case_dir,
output_dir=case.output_dir,
frame_window_source="manual",
target_frame_id=args.target_frame_id,
requested_frame_index_start=args.frame_index_start,
requested_frame_index_end=args.frame_index_end,
requested_frame_id_start=requested_frame_id_start,
requested_frame_id_end=requested_frame_id_end,
)
prepared_cases.append(prepared_case)
continue
if not args.use_issue_frame_window:
prepared_cases.append(prepared_case)
continue
item = issue_item_lookup.get(case.issue_id)
if item is None:
if args.missing_issue_frame_policy == "skip":
skipped_results.append(
build_case_result(
prepared_case,
status="skipped_missing_issue_record",
detail="issue id not found in issue_json",
)
)
continue
prepared_case = InferenceCase(
issue_id=case.issue_id,
issue_dir=case.issue_dir,
case_dir=case.case_dir,
camera4_bin=case.camera4_bin,
relative_case_dir=case.relative_case_dir,
output_dir=case.output_dir,
frame_window_source="full_missing_issue_record",
)
prepared_cases.append(prepared_case)
continue
target_frame, target_frame_error = parse_target_frame(item.get("问题发生frameid"))
if target_frame is None:
if args.missing_issue_frame_policy == "skip":
skipped_results.append(
build_case_result(
prepared_case,
status="skipped_missing_issue_frame",
detail=target_frame_error,
)
)
continue
prepared_case = InferenceCase(
issue_id=case.issue_id,
issue_dir=case.issue_dir,
case_dir=case.case_dir,
camera4_bin=case.camera4_bin,
relative_case_dir=case.relative_case_dir,
output_dir=case.output_dir,
frame_window_source=(
"full_missing_issue_frame"
if target_frame_error.startswith("missing ")
else "full_invalid_issue_frame"
),
)
prepared_cases.append(prepared_case)
continue
if target_frame.camera not in {"camera4", "any"}:
if args.missing_issue_frame_policy == "skip":
skipped_results.append(
build_case_result(
prepared_case,
status="skipped_unsupported_issue_camera",
detail=f"unsupported target camera for window inference: {target_frame.camera}",
)
)
continue
prepared_case = InferenceCase(
issue_id=case.issue_id,
issue_dir=case.issue_dir,
case_dir=case.case_dir,
camera4_bin=case.camera4_bin,
relative_case_dir=case.relative_case_dir,
output_dir=case.output_dir,
frame_window_source="full_unsupported_issue_camera",
target_frame_text=target_frame.raw_text,
)
prepared_cases.append(prepared_case)
continue
prepared_case = InferenceCase(
issue_id=case.issue_id,
issue_dir=case.issue_dir,
case_dir=case.case_dir,
camera4_bin=case.camera4_bin,
relative_case_dir=case.relative_case_dir,
output_dir=case.output_dir,
frame_window_source="issue_frame",
target_frame_text=target_frame.raw_text,
target_frame_id=target_frame.frame_id,
requested_frame_id_start=max(0, target_frame.frame_id - args.frame_before),
requested_frame_id_end=target_frame.frame_id + args.frame_after,
)
prepared_cases.append(prepared_case)
return prepared_cases, skipped_results
def has_existing_outputs(output_dir: Path) -> bool:
if (output_dir / "predictions_merged.json").is_file():
return True
merge_json_dir = output_dir / "predictions" / "merge"
if merge_json_dir.is_dir():
for child in merge_json_dir.iterdir():
if child.is_file():
return True
return False
def build_command(case: InferenceCase, args: argparse.Namespace) -> list[str]:
command = [
str(Path(args.python_bin).resolve()),
str(Path(args.inference_script).resolve()),
"--video-case-dir",
str(case.camera4_bin),
"--output-dir",
str(case.output_dir),
"--video-stride",
str(args.video_stride),
]
if args.max_images > 0:
command.extend(["--max-images", str(args.max_images)])
if case.target_frame_id is not None:
command.extend(["--target-frame-id", str(case.target_frame_id)])
if case.requested_frame_index_start is not None:
command.extend(["--frame-index-start", str(case.requested_frame_index_start)])
if case.requested_frame_index_end is not None:
command.extend(["--frame-index-end", str(case.requested_frame_index_end)])
if case.requested_frame_id_start is not None:
command.extend(["--frame-id-start", str(case.requested_frame_id_start)])
if case.requested_frame_id_end is not None:
command.extend(["--frame-id-end", str(case.requested_frame_id_end)])
if args.exported_model:
command.extend(["--exported-model", args.exported_model])
if args.device:
command.extend(["--device", args.device])
if args.providers:
command.extend(["--providers", *args.providers])
if args.enable_attr:
command.append("--enable-attr")
if args.enable_cross_class_merge_prior:
command.append("--enable-cross-class-merge-prior")
if args.save_aggregate_predictions:
command.append("--save-aggregate-predictions")
for extra_arg in args.inference_arg:
command.append(extra_arg)
return command
def write_case_status_files(case: InferenceCase, command: list[str], log_text: str, dry_run: bool) -> str:
status_dir = case.output_dir / "_status"
if dry_run:
return str(status_dir / "inference.log")
ensure_dir(status_dir, dry_run=False)
(status_dir / "command.txt").write_text(" ".join(command) + "\n", encoding="utf-8")
log_path = status_dir / "inference.log"
log_path.write_text(log_text, encoding="utf-8")
return str(log_path)
def summarize_process_output(completed: subprocess.CompletedProcess[str]) -> str:
stdout = completed.stdout.strip()
stderr = completed.stderr.strip()
if completed.returncode == 0:
if stdout:
return stdout.splitlines()[-1]
return "inference completed"
if stderr:
return stderr.splitlines()[-1]
if stdout:
return stdout.splitlines()[-1]
return f"inference failed with return code {completed.returncode}"
def run_case(case: InferenceCase, args: argparse.Namespace) -> CaseResult:
command = build_command(case, args)
if args.skip_existing and has_existing_outputs(case.output_dir):
return build_case_result(
case,
status="skipped_existing",
detail="existing inference outputs found",
command=command,
log_path=None,
)
if args.dry_run:
return build_case_result(
case,
status="planned",
detail="would run inference",
command=command,
log_path=str(case.output_dir / "_status" / "inference.log"),
)
ensure_dir(case.output_dir, dry_run=False)
completed = subprocess.run(
command,
cwd=str(ROOT),
check=False,
capture_output=True,
text=True,
encoding="utf-8",
)
combined_log = "\n".join(
part for part in (completed.stdout.strip(), completed.stderr.strip()) if part
)
log_path = write_case_status_files(case, command, combined_log + ("\n" if combined_log else ""), dry_run=False)
status = "success" if completed.returncode == 0 else "failed"
detail = summarize_process_output(completed)
return build_case_result(
case,
status=status,
detail=detail,
command=command,
log_path=log_path,
)
def build_manifest(
args: argparse.Namespace,
download_root: Path,
output_root: Path,
discovered_cases: list[InferenceCase],
results: list[CaseResult],
) -> dict:
summary: dict[str, int] = {}
for result in results:
summary[result.status] = summary.get(result.status, 0) + 1
return {
"generated_at": datetime.now().astimezone().isoformat(timespec="seconds"),
"download_root": str(download_root),
"output_root": str(output_root),
"python_bin": str(Path(args.python_bin).resolve()),
"inference_script": str(Path(args.inference_script).resolve()),
"dry_run": args.dry_run,
"skip_existing": args.skip_existing,
"issue_filter": args.issue_ids or [],
"issue_json": args.issue_json,
"use_issue_frame_window": args.use_issue_frame_window,
"missing_issue_frame_policy": args.missing_issue_frame_policy,
"video_stride": args.video_stride,
"max_images": args.max_images,
"frame_index_start": args.frame_index_start,
"frame_index_end": args.frame_index_end,
"frame_id_start": args.frame_id_start,
"frame_id_end": args.frame_id_end,
"target_frame_id": args.target_frame_id,
"frame_before": args.frame_before,
"frame_after": args.frame_after,
"exported_model": args.exported_model,
"device": args.device,
"providers": args.providers or [],
"enable_attr": args.enable_attr,
"enable_cross_class_merge_prior": args.enable_cross_class_merge_prior,
"save_aggregate_predictions": args.save_aggregate_predictions,
"inference_args": args.inference_arg,
"total_cases": len(discovered_cases),
"summary": summary,
"cases": [result.to_dict() for result in results],
}
def print_summary(manifest: dict) -> None:
print(f"download_root: {manifest['download_root']}")
print(f"output_root: {manifest['output_root']}")
print(f"dry_run: {manifest['dry_run']}")
if manifest["use_issue_frame_window"]:
print(
"issue_frame_window: "
f"enabled issue_json={manifest['issue_json']} "
f"before={manifest['frame_before']} after={manifest['frame_after']} "
f"missing_policy={manifest['missing_issue_frame_policy']}"
)
elif (
manifest["target_frame_id"] is not None
or manifest["frame_id_start"] is not None
or manifest["frame_id_end"] is not None
or manifest["frame_index_start"] is not None
or manifest["frame_index_end"] is not None
):
print(
"manual_frame_window: "
f"target_frame_id={manifest['target_frame_id']} "
f"frame_id=[{manifest['frame_id_start'] if manifest['frame_id_start'] is not None else '-inf'}, "
f"{manifest['frame_id_end'] if manifest['frame_id_end'] is not None else '+inf'}] "
f"frame_index=[{manifest['frame_index_start'] if manifest['frame_index_start'] is not None else '-inf'}, "
f"{manifest['frame_index_end'] if manifest['frame_index_end'] is not None else '+inf'}]"
)
print(f"total_cases: {manifest['total_cases']}")
for status, count in sorted(manifest["summary"].items()):
print(f"{status}: {count}")
def main() -> int:
args = parse_args()
download_root = Path(args.download_root).resolve()
output_root = Path(args.output_root).resolve()
manifest_path = (
Path(args.manifest_path).resolve()
if args.manifest_path
else output_root / "inference_manifest.json"
)
issue_filter = set(args.issue_ids) if args.issue_ids else None
cases = discover_cases(download_root, output_root, issue_filter)
if not cases:
raise FileNotFoundError(f"No */sigmastar.1/camera4.bin cases found under {download_root}")
issue_item_lookup: dict[int, dict] = {}
if args.use_issue_frame_window:
issue_item_lookup = build_issue_item_lookup(Path(args.issue_json).resolve())
prepared_cases, skipped_results = apply_frame_windows(cases, args, issue_item_lookup)
log_progress(
"cases discovered: "
f"total={len(cases)} runnable={len(prepared_cases)} skipped_precheck={len(skipped_results)}"
)
for skipped_result in skipped_results:
log_progress(
f"skip issue_{skipped_result.issue_id} {compact_text(skipped_result.relative_case_dir)}: "
f"{skipped_result.status} ({compact_text(skipped_result.detail, max_len=72)})"
)
ensure_dir(output_root, dry_run=args.dry_run)
results = list(skipped_results)
total_prepared_cases = len(prepared_cases)
for index, case in enumerate(prepared_cases, start=1):
log_progress(
f"[{index}/{total_prepared_cases}] issue_{case.issue_id} "
f"{compact_text(case.relative_case_dir)}: start (window={case.frame_window_source})"
)
case_result = run_case(case, args)
results.append(case_result)
log_progress(
f"[{index}/{total_prepared_cases}] issue_{case.issue_id} "
f"{compact_text(case.relative_case_dir)}: "
f"{case_result.status} ({compact_text(case_result.detail, max_len=72)})"
)
manifest = build_manifest(args, download_root, output_root, cases, results)
if not args.dry_run:
ensure_dir(manifest_path.parent, dry_run=False)
manifest_path.write_text(
json.dumps(manifest, ensure_ascii=False, indent=2) + "\n",
encoding="utf-8",
)
print_summary(manifest)
if args.dry_run:
print(f"manifest (not written in dry-run): {manifest_path}")
else:
print(f"manifest: {manifest_path}")
return 0
if __name__ == "__main__":
sys.exit(main())

View File

@@ -0,0 +1,193 @@
#!/usr/bin/env bash
set -euo pipefail
SCRIPT_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
PROJECT_ROOT=$(cd "${SCRIPT_DIR}/../.." && pwd)
DOWNLOAD_ROOT=${DOWNLOAD_ROOT:-/data1/dongying/Mono3d/G1Q3/feishu_project/downloaded_issue_data}
OUTPUT_ROOT=${OUTPUT_ROOT:-/data1/dongying/Mono3d/G1Q3/feishu_project/inference_issue_data}
MANIFEST_PATH=${MANIFEST_PATH:-"${OUTPUT_ROOT}/inference_manifest.json"}
PYTHON_BIN=${PYTHON_BIN:-/deeplearning_team/ydong/dongying/miniconda/envs/dev/bin/python}
INFERENCE_SCRIPT=${INFERENCE_SCRIPT:-"${PROJECT_ROOT}/tools/model_inference/core/run_two_roi_exported_onnx_infer.py"}
ISSUE_TRACKING_SCRIPT=${ISSUE_TRACKING_SCRIPT:-"${PROJECT_ROOT}/tools/feishu_project/run_issue_data_tracking.sh"}
ISSUE_JSON=${ISSUE_JSON:-}
VIDEO_STRIDE=${VIDEO_STRIDE:-1}
MAX_IMAGES=${MAX_IMAGES:-0}
FRAME_INDEX_START=${FRAME_INDEX_START:-}
FRAME_INDEX_END=${FRAME_INDEX_END:-}
FRAME_ID_START=${FRAME_ID_START:-}
FRAME_ID_END=${FRAME_ID_END:-}
TARGET_FRAME_ID=${TARGET_FRAME_ID:-}
FRAME_BEFORE=${FRAME_BEFORE:-100}
FRAME_AFTER=${FRAME_AFTER:-100}
USE_ISSUE_FRAME_WINDOW=${USE_ISSUE_FRAME_WINDOW:-0}
MISSING_ISSUE_FRAME_POLICY=${MISSING_ISSUE_FRAME_POLICY:-full}
EXPORTED_MODEL=${EXPORTED_MODEL:-${PROJECT_ROOT}/runs/export/train_mono3d_two_roi_20260416-raw_no_edge/merged_model.torchscript}
DEVICE=${DEVICE:-}
PROVIDERS=${PROVIDERS:-}
ENABLE_ATTR=${ENABLE_ATTR:-0}
ENABLE_CROSS_CLASS_MERGE_PRIOR=${ENABLE_CROSS_CLASS_MERGE_PRIOR:-1}
SAVE_AGGREGATE_PREDICTIONS=${SAVE_AGGREGATE_PREDICTIONS:-1}
SKIP_EXISTING=${SKIP_EXISTING:-0}
DRY_RUN=${DRY_RUN:-0}
ENABLE_TRACKING=${ENABLE_TRACKING:-0}
INFERENCE_EXTRA_ARGS=${INFERENCE_EXTRA_ARGS:-}
resolve_tracking_model_version() {
if [[ -n "${TRACKING_MODEL_VERSION:-}" ]]; then
printf '%s\n' "${TRACKING_MODEL_VERSION}"
return 0
fi
if [[ -n "${EXPORTED_MODEL}" ]] && [[ "${EXPORTED_MODEL}" =~ ([0-9]{8}) ]]; then
printf '%s\n' "${BASH_REMATCH[1]}"
return 0
fi
return 1
}
FORWARD_ARGS=()
ISSUE_ID_ARGS=()
while (($# > 0)); do
case "$1" in
--issue-id)
if (($# < 2)); then
echo "Error: --issue-id requires a value" >&2
exit 1
fi
ISSUE_ID_ARGS+=("$1" "$2")
FORWARD_ARGS+=("$1" "$2")
shift 2
;;
--issue-id=*)
ISSUE_ID_ARGS+=("$1")
FORWARD_ARGS+=("$1")
shift
;;
*)
FORWARD_ARGS+=("$1")
shift
;;
esac
done
CMD=(
"${PYTHON_BIN}" "${PROJECT_ROOT}/tools/feishu_project/run_issue_data_inference.py"
--download-root "${DOWNLOAD_ROOT}"
--output-root "${OUTPUT_ROOT}"
--manifest-path "${MANIFEST_PATH}"
--python-bin "${PYTHON_BIN}"
--inference-script "${INFERENCE_SCRIPT}"
--video-stride "${VIDEO_STRIDE}"
)
if [[ -n "${ISSUE_JSON}" ]]; then
CMD+=(--issue-json "${ISSUE_JSON}")
fi
if [[ "${MAX_IMAGES}" != "0" ]]; then
CMD+=(--max-images "${MAX_IMAGES}")
fi
if [[ -n "${FRAME_INDEX_START}" ]]; then
CMD+=(--frame-index-start "${FRAME_INDEX_START}")
fi
if [[ -n "${FRAME_INDEX_END}" ]]; then
CMD+=(--frame-index-end "${FRAME_INDEX_END}")
fi
if [[ -n "${FRAME_ID_START}" ]]; then
CMD+=(--frame-id-start "${FRAME_ID_START}")
fi
if [[ -n "${FRAME_ID_END}" ]]; then
CMD+=(--frame-id-end "${FRAME_ID_END}")
fi
if [[ -n "${TARGET_FRAME_ID}" ]]; then
CMD+=(--target-frame-id "${TARGET_FRAME_ID}")
fi
if [[ "${USE_ISSUE_FRAME_WINDOW}" != "1" && "${FRAME_BEFORE}" != "100" ]]; then
CMD+=(--frame-before "${FRAME_BEFORE}")
fi
if [[ "${USE_ISSUE_FRAME_WINDOW}" != "1" && "${FRAME_AFTER}" != "100" ]]; then
CMD+=(--frame-after "${FRAME_AFTER}")
fi
if [[ "${USE_ISSUE_FRAME_WINDOW}" == "1" ]]; then
CMD+=(--use-issue-frame-window --frame-before "${FRAME_BEFORE}" --frame-after "${FRAME_AFTER}" --missing-issue-frame-policy "${MISSING_ISSUE_FRAME_POLICY}")
fi
if [[ -n "${EXPORTED_MODEL}" ]]; then
CMD+=(--exported-model "${EXPORTED_MODEL}")
fi
if [[ -n "${DEVICE}" ]]; then
CMD+=(--device "${DEVICE}")
fi
if [[ -n "${PROVIDERS}" ]]; then
# shellcheck disable=SC2206
PROVIDER_ARR=(${PROVIDERS})
CMD+=(--providers "${PROVIDER_ARR[@]}")
fi
if [[ "${ENABLE_ATTR}" == "1" ]]; then
CMD+=(--enable-attr)
fi
if [[ "${ENABLE_CROSS_CLASS_MERGE_PRIOR}" == "1" ]]; then
CMD+=(--enable-cross-class-merge-prior)
fi
if [[ "${SAVE_AGGREGATE_PREDICTIONS}" == "1" ]]; then
CMD+=(--save-aggregate-predictions)
fi
if [[ "${SKIP_EXISTING}" == "1" ]]; then
CMD+=(--skip-existing)
fi
if [[ "${DRY_RUN}" == "1" ]]; then
CMD+=(--dry-run)
fi
if [[ -n "${INFERENCE_EXTRA_ARGS}" ]]; then
# shellcheck disable=SC2206
EXTRA_ARR=(${INFERENCE_EXTRA_ARGS})
for arg in "${EXTRA_ARR[@]}"; do
CMD+=("--inference-arg=${arg}")
done
fi
CMD+=("${FORWARD_ARGS[@]}")
"${CMD[@]}"
if [[ "${ENABLE_TRACKING}" != "1" ]]; then
exit 0
fi
if [[ "${DRY_RUN}" == "1" ]]; then
echo "Skipping tracking because DRY_RUN=1"
exit 0
fi
TRACKING_MODEL_VERSION_RESOLVED=${TRACKING_MODEL_VERSION:-}
if [[ -z "${TRACKING_MODEL_VERSION_RESOLVED}" ]]; then
TRACKING_MODEL_VERSION_RESOLVED=$(resolve_tracking_model_version || true)
fi
if [[ -n "${TRACKING_MODEL_VERSION_RESOLVED}" ]]; then
echo "Running tracking with model version: ${TRACKING_MODEL_VERSION_RESOLVED}"
TRACKING_MODEL_VERSION="${TRACKING_MODEL_VERSION_RESOLVED}" \
PYTHON_BIN="${PYTHON_BIN}" \
bash "${ISSUE_TRACKING_SCRIPT}" "${OUTPUT_ROOT}" "${ISSUE_ID_ARGS[@]}"
else
echo "Running tracking without an explicit model version override"
PYTHON_BIN="${PYTHON_BIN}" \
bash "${ISSUE_TRACKING_SCRIPT}" "${OUTPUT_ROOT}" "${ISSUE_ID_ARGS[@]}"
fi

View File

@@ -0,0 +1,360 @@
#!/usr/bin/env bash
set -euo pipefail
SCRIPT_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
PROJECT_ROOT=$(cd "${SCRIPT_DIR}/../.." && pwd)
export PYTHONPATH="${PROJECT_ROOT}:${PYTHONPATH:-}"
RESULTS_ROOT=${RESULTS_ROOT:-/data1/dongying/Mono3d/G1Q3/feishu_project/inference_issue_data}
OUTPUT_ROOT=${OUTPUT_ROOT:-}
OUTPUT_DIR_NAME=${OUTPUT_DIR_NAME:-temporal_observation}
PYTHON_BIN=${PYTHON_BIN:-/deeplearning_team/ydong/dongying/miniconda/envs/dev/bin/python}
TEMPORAL_SCRIPT=${TEMPORAL_SCRIPT:-"${PROJECT_ROOT}/tools/temporal_analysis/evaluate_temporal_stability.py"}
TRACKING_JSON_NAME=${TRACKING_JSON_NAME:-merge.json}
TEMPORAL_MIN_LENGTH=${TEMPORAL_MIN_LENGTH:-3}
TEMPORAL_CLASS_ID=${TEMPORAL_CLASS_ID:-}
TEMPORAL_TRACK_IDS=${TEMPORAL_TRACK_IDS:-4}
TEMPORAL_FRAME_ID_START=${TEMPORAL_FRAME_ID_START:-86421}
TEMPORAL_FRAME_ID_END=${TEMPORAL_FRAME_ID_END:-86441}
TEMPORAL_PLOTS=${TEMPORAL_PLOTS:-0}
TEMPORAL_EXPORT_SERIES=${TEMPORAL_EXPORT_SERIES:-1}
TEMPORAL_FOCUS_TRACK_PLOTS=${TEMPORAL_FOCUS_TRACK_PLOTS:-1}
TEMPORAL_PREFER_EGO=${TEMPORAL_PREFER_EGO:-1}
TEMPORAL_X_AXIS=${TEMPORAL_X_AXIS:-frame_id}
TEMPORAL_HEADING_SOURCE=${TEMPORAL_HEADING_SOURCE:-camera_reg}
TARGET_PATH="/data1/dongying/Mono3d/G1Q3/feishu_project/inference_issue_data/issue_6974203002/pdcl_01/20260417112730/2026-04-17_11-29-29_voice-HMI二轮车位置不正确"
ISSUE_IDS=()
CLI_TRACK_IDS=()
CLI_CLASS_ID=""
CLI_FRAME_ID_START=""
CLI_FRAME_ID_END=""
while (($# > 0)); do
case "$1" in
--issue-id)
if (($# < 2)); then
echo "Error: --issue-id requires a value" >&2
exit 1
fi
ISSUE_IDS+=("$2")
shift 2
;;
--issue-id=*)
ISSUE_IDS+=("${1#*=}")
shift
;;
--track-id)
if (($# < 2)); then
echo "Error: --track-id requires a value" >&2
exit 1
fi
CLI_TRACK_IDS+=("$2")
shift 2
;;
--track-id=*)
CLI_TRACK_IDS+=("${1#*=}")
shift
;;
--class-id)
if (($# < 2)); then
echo "Error: --class-id requires a value" >&2
exit 1
fi
CLI_CLASS_ID="$2"
shift 2
;;
--class-id=*)
CLI_CLASS_ID="${1#*=}"
shift
;;
--frame-id-start)
if (($# < 2)); then
echo "Error: --frame-id-start requires a value" >&2
exit 1
fi
CLI_FRAME_ID_START="$2"
shift 2
;;
--frame-id-start=*)
CLI_FRAME_ID_START="${1#*=}"
shift
;;
--frame-id-end)
if (($# < 2)); then
echo "Error: --frame-id-end requires a value" >&2
exit 1
fi
CLI_FRAME_ID_END="$2"
shift 2
;;
--frame-id-end=*)
CLI_FRAME_ID_END="${1#*=}"
shift
;;
-*)
echo "Error: unsupported option: $1" >&2
exit 1
;;
*)
if [[ -n "${TARGET_PATH}" ]]; then
echo "Error: multiple target paths provided: ${TARGET_PATH} and $1" >&2
exit 1
fi
TARGET_PATH="$1"
shift
;;
esac
done
if [[ -z "${TARGET_PATH}" ]]; then
TARGET_PATH="${RESULTS_ROOT}"
fi
if [[ ! -e "${TARGET_PATH}" ]]; then
echo "Error: target path does not exist: ${TARGET_PATH}" >&2
exit 1
fi
resolve_abs_path() {
local target_path="$1"
if [[ -d "${target_path}" ]]; then
(
cd "${target_path}"
pwd -P
)
return 0
fi
local parent_dir
parent_dir=$(
cd "$(dirname "${target_path}")"
pwd -P
)
printf '%s/%s\n' "${parent_dir}" "$(basename "${target_path}")"
}
RESULTS_ROOT_ABS="$(resolve_abs_path "${RESULTS_ROOT}")"
is_case_dir() {
local dir_path="$1"
[[ -d "${dir_path}" ]] && [[ -f "${dir_path}/${TRACKING_JSON_NAME}" ]]
}
derive_output_dir() {
local case_dir="$1"
local case_abs
local rel_case_dir
case_abs="$(resolve_abs_path "${case_dir}")"
if [[ -n "${OUTPUT_ROOT}" ]]; then
if [[ "${case_abs}" == "${RESULTS_ROOT_ABS}" ]]; then
printf '%s\n' "${OUTPUT_ROOT}"
return 0
fi
if [[ "${case_abs}" == "${RESULTS_ROOT_ABS}"/* ]]; then
rel_case_dir="${case_abs#"${RESULTS_ROOT_ABS}/"}"
printf '%s/%s\n' "${OUTPUT_ROOT%/}" "${rel_case_dir}"
return 0
fi
printf '%s/%s\n' "${OUTPUT_ROOT%/}" "$(basename "${case_abs}")"
return 0
fi
printf '%s/%s\n' "${case_abs}" "${OUTPUT_DIR_NAME}"
}
build_track_id_args() {
local -n out_ref=$1
out_ref=()
if [[ "${#CLI_TRACK_IDS[@]}" -gt 0 ]]; then
for track_id in "${CLI_TRACK_IDS[@]}"; do
out_ref+=(--track-ids "${track_id}")
done
return 0
fi
if [[ -n "${TEMPORAL_TRACK_IDS}" ]]; then
# shellcheck disable=SC2206
local track_id_arr=(${TEMPORAL_TRACK_IDS})
for track_id in "${track_id_arr[@]}"; do
out_ref+=(--track-ids "${track_id}")
done
fi
}
run_single_case() {
local case_dir="$1"
local tracking_json="${case_dir}/${TRACKING_JSON_NAME}"
local output_dir
local cmd
local track_id_args
local class_id_value=""
local frame_id_start_value=""
local frame_id_end_value=""
if [[ ! -f "${tracking_json}" ]]; then
echo "[ERROR] ${TRACKING_JSON_NAME} not found in case directory: ${case_dir}" >&2
return 1
fi
output_dir="$(derive_output_dir "${case_dir}")"
build_track_id_args track_id_args
if [[ -n "${CLI_CLASS_ID}" ]]; then
class_id_value="${CLI_CLASS_ID}"
elif [[ -n "${TEMPORAL_CLASS_ID}" ]]; then
class_id_value="${TEMPORAL_CLASS_ID}"
fi
if [[ -n "${CLI_FRAME_ID_START}" ]]; then
frame_id_start_value="${CLI_FRAME_ID_START}"
elif [[ -n "${TEMPORAL_FRAME_ID_START}" ]]; then
frame_id_start_value="${TEMPORAL_FRAME_ID_START}"
fi
if [[ -n "${CLI_FRAME_ID_END}" ]]; then
frame_id_end_value="${CLI_FRAME_ID_END}"
elif [[ -n "${TEMPORAL_FRAME_ID_END}" ]]; then
frame_id_end_value="${TEMPORAL_FRAME_ID_END}"
fi
cmd=(
"${PYTHON_BIN}" "${TEMPORAL_SCRIPT}"
--input "${tracking_json}"
--output "${output_dir}/stability_report.json"
--min-length "${TEMPORAL_MIN_LENGTH}"
--x-axis "${TEMPORAL_X_AXIS}"
--heading-source "${TEMPORAL_HEADING_SOURCE}"
)
if [[ -n "${class_id_value}" ]]; then
cmd+=(--class-id "${class_id_value}")
fi
if [[ -n "${frame_id_start_value}" ]]; then
cmd+=(--frame-id-start "${frame_id_start_value}")
fi
if [[ -n "${frame_id_end_value}" ]]; then
cmd+=(--frame-id-end "${frame_id_end_value}")
fi
if [[ "${#track_id_args[@]}" -gt 0 ]]; then
cmd+=("${track_id_args[@]}")
fi
if [[ "${TEMPORAL_PLOTS}" == "1" ]]; then
cmd+=(--plots)
fi
if [[ "${TEMPORAL_EXPORT_SERIES}" == "1" ]]; then
cmd+=(--export-series)
fi
if [[ "${TEMPORAL_FOCUS_TRACK_PLOTS}" == "1" ]]; then
cmd+=(--focus-track-plots)
fi
if [[ "${TEMPORAL_PREFER_EGO}" == "1" ]]; then
cmd+=(--prefer-ego)
else
cmd+=(--no-prefer-ego)
fi
echo ""
echo "######################################################################"
echo "# Feishu issue-data temporal observation"
echo "######################################################################"
echo "Case : ${case_dir}"
echo "Track : ${tracking_json}"
echo "Output : ${output_dir}"
if [[ -n "${class_id_value}" ]]; then
echo "Class : ${class_id_value}"
fi
if [[ -n "${frame_id_start_value}" || -n "${frame_id_end_value}" ]]; then
echo "Frame ID Range: [${frame_id_start_value:-"-inf"}, ${frame_id_end_value:-"+inf"}]"
fi
echo "Heading Source: ${TEMPORAL_HEADING_SOURCE}"
if [[ "${#CLI_TRACK_IDS[@]}" -gt 0 ]]; then
echo "Track IDs: ${CLI_TRACK_IDS[*]}"
elif [[ -n "${TEMPORAL_TRACK_IDS}" ]]; then
echo "Track IDs: ${TEMPORAL_TRACK_IDS}"
fi
"${cmd[@]}"
}
run_batch_root() {
local batch_root="$1"
local total_cases=0
local success_cases=0
local failed_cases=0
while IFS= read -r -d '' tracking_json; do
local case_dir
case_dir="$(dirname "${tracking_json}")"
((total_cases += 1))
if run_single_case "${case_dir}"; then
((success_cases += 1))
else
((failed_cases += 1))
printf '[FAIL] case=%s\n' "${case_dir}" >&2
fi
done < <(
find "${batch_root}" -type f -name "${TRACKING_JSON_NAME}" -print0 | sort -z
)
if [[ "${total_cases}" -eq 0 ]]; then
echo "[ERROR] No ${TRACKING_JSON_NAME} files were found under: ${batch_root}" >&2
return 1
fi
echo ""
printf '[DONE] cases=%d success=%d failed=%d\n' \
"${total_cases}" "${success_cases}" "${failed_cases}"
[[ "${failed_cases}" -eq 0 ]]
}
if [[ "${#ISSUE_IDS[@]}" -eq 0 ]]; then
if is_case_dir "${TARGET_PATH}"; then
run_single_case "${TARGET_PATH}"
elif [[ -d "${TARGET_PATH}" ]]; then
run_batch_root "${TARGET_PATH}"
else
echo "Error: unsupported target path: ${TARGET_PATH}" >&2
exit 1
fi
exit 0
fi
total_issues=0
success_issues=0
failed_issues=0
for issue_id in "${ISSUE_IDS[@]}"; do
issue_root="${TARGET_PATH}/issue_${issue_id}"
((total_issues += 1))
if [[ ! -d "${issue_root}" ]]; then
echo "[FAIL] issue_${issue_id}: not found under ${TARGET_PATH}" >&2
((failed_issues += 1))
continue
fi
if run_batch_root "${issue_root}"; then
((success_issues += 1))
else
((failed_issues += 1))
echo "[FAIL] issue_${issue_id}: temporal observation failed" >&2
fi
done
echo ""
printf '[DONE] issues=%d success=%d failed=%d\n' \
"${total_issues}" "${success_issues}" "${failed_issues}"
[[ "${failed_issues}" -eq 0 ]]

View File

@@ -0,0 +1,347 @@
#!/usr/bin/env bash
set -euo pipefail
SCRIPT_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
PROJECT_ROOT=$(cd "${SCRIPT_DIR}/../.." && pwd)
RESULTS_ROOT=${RESULTS_ROOT:-/data1/dongying/Mono3d/G1Q3/feishu_project/inference_issue_data}
PYTHON_BIN=${PYTHON_BIN:-/deeplearning_team/ydong/dongying/miniconda/envs/dev/bin/python}
TRACKING_LAUNCHER=${TRACKING_LAUNCHER:-"${PROJECT_ROOT}/tools/temporal_analysis/track_objects_exported_onnx_infer_case.sh"}
ISSUE_CONVERT_SCRIPT=${ISSUE_CONVERT_SCRIPT:-"${PROJECT_ROOT}/tools/feishu_project/run_issue_data_convert_tracking.sh"}
ISSUE_VISUALIZE_SCRIPT=${ISSUE_VISUALIZE_SCRIPT:-"${PROJECT_ROOT}/tools/feishu_project/run_issue_data_visualize_tracking.sh"}
ISSUE_TEMPORAL_SCRIPT=${ISSUE_TEMPORAL_SCRIPT:-"${PROJECT_ROOT}/tools/feishu_project/run_issue_data_temporal_observe.sh"}
TRACK_CLASSES=${TRACK_CLASSES:-0 1 2 3 4 5 6 7 8 9 10 11 12 17 18 19}
IOU_THRESH=${IOU_THRESH:-0.3}
MAX_AGE=${MAX_AGE:-5}
MIN_HITS=${MIN_HITS:-1}
DIST_THRESH=${DIST_THRESH:-100}
MAX_3D_DISTANCE=${MAX_3D_DISTANCE:-10.0}
MAX_FRAMES=${MAX_FRAMES:-}
MERGE_OUTPUT_NAME=${MERGE_OUTPUT_NAME:-combined_tracking.json}
FILE_PATTERN=${FILE_PATTERN:-*.json}
ENABLE_USE_3D=${ENABLE_USE_3D:-0}
ENABLE_CONVERT=${ENABLE_CONVERT:-0}
ENABLE_VISUALIZE=${ENABLE_VISUALIZE:-1}
ENABLE_TEMPORAL_ANALYSIS=${ENABLE_TEMPORAL_ANALYSIS:-0}
CONVERT_OUTPUT_DIR_NAME=${CONVERT_OUTPUT_DIR_NAME:-objectlist}
CONVERT_TRACKING_JSON_NAME=${CONVERT_TRACKING_JSON_NAME:-merge.json}
CONVERT_CAM_ID=${CONVERT_CAM_ID:-}
VIS_DOWNLOAD_ROOT=${VIS_DOWNLOAD_ROOT:-/data1/dongying/Mono3d/G1Q3/feishu_project/downloaded_issue_data}
VIS_OUTPUT_ROOT=${VIS_OUTPUT_ROOT:-}
VIS_OUTPUT_DIR_NAME=${VIS_OUTPUT_DIR_NAME:-tracking_vis_raw}
VIS_TRACKING_JSON_NAME=${VIS_TRACKING_JSON_NAME:-merge.json}
VIS_MAX_FRAMES=${VIS_MAX_FRAMES:-}
VIS_CLASS_ID=${VIS_CLASS_ID:-}
VIS_TRACK_IDS=${VIS_TRACK_IDS:-}
VIS_SHOW_TRAJECTORY=${VIS_SHOW_TRAJECTORY:-0}
TEMPORAL_OUTPUT_ROOT=${TEMPORAL_OUTPUT_ROOT:-}
TEMPORAL_OUTPUT_DIR_NAME=${TEMPORAL_OUTPUT_DIR_NAME:-temporal_observation}
TEMPORAL_JSON_NAME=${TEMPORAL_JSON_NAME:-merge.json}
TEMPORAL_MIN_LENGTH=${TEMPORAL_MIN_LENGTH:-3}
TEMPORAL_TRACK_IDS=${TEMPORAL_TRACK_IDS:-}
TEMPORAL_CLASS_ID=${TEMPORAL_CLASS_ID:-}
TEMPORAL_FRAME_ID_START=${TEMPORAL_FRAME_ID_START:-}
TEMPORAL_FRAME_ID_END=${TEMPORAL_FRAME_ID_END:-}
TEMPORAL_PLOTS=${TEMPORAL_PLOTS:-0}
TEMPORAL_EXPORT_SERIES=${TEMPORAL_EXPORT_SERIES:-1}
TEMPORAL_FOCUS_TRACK_PLOTS=${TEMPORAL_FOCUS_TRACK_PLOTS:-1}
TEMPORAL_PREFER_EGO=${TEMPORAL_PREFER_EGO:-1}
TEMPORAL_X_AXIS=${TEMPORAL_X_AXIS:-frame_id}
TEMPORAL_HEADING_SOURCE=${TEMPORAL_HEADING_SOURCE:-camera_reg}
TRACKING_MODEL_VERSION=${TRACKING_MODEL_VERSION:-20260416}
TARGET_PATH=${TARGET_PATH:-}
ISSUE_IDS=()
while (($# > 0)); do
case "$1" in
--issue-id)
if (($# < 2)); then
echo "Error: --issue-id requires a value" >&2
exit 1
fi
ISSUE_IDS+=("$2")
shift 2
;;
--issue-id=*)
ISSUE_IDS+=("${1#*=}")
shift
;;
-*)
echo "Error: unsupported option: $1" >&2
exit 1
;;
*)
if [[ -n "${TARGET_PATH}" ]]; then
echo "Error: multiple target paths provided: ${TARGET_PATH} and $1" >&2
exit 1
fi
TARGET_PATH="$1"
shift
;;
esac
done
if [[ -z "${TARGET_PATH}" ]]; then
TARGET_PATH="${RESULTS_ROOT}"
fi
if [[ ! -d "${TARGET_PATH}" ]]; then
echo "Error: target directory does not exist: ${TARGET_PATH}" >&2
exit 1
fi
run_tracking_wrapper() {
local target_path="$1"
echo ""
echo "######################################################################"
echo "# Feishu issue-data tracking"
echo "######################################################################"
echo "Target path: ${target_path}"
if [[ -n "${TRACKING_MODEL_VERSION}" ]]; then
echo "Model version: ${TRACKING_MODEL_VERSION}"
fi
if [[ -n "${TRACKING_MODEL_VERSION}" ]]; then
MODEL_VERSION="${TRACKING_MODEL_VERSION}" \
PYTHON_BIN="${PYTHON_BIN}" \
RESULTS_ROOT="${target_path}" \
TRACK_CLASSES="${TRACK_CLASSES}" \
IOU_THRESH="${IOU_THRESH}" \
MAX_AGE="${MAX_AGE}" \
MIN_HITS="${MIN_HITS}" \
DIST_THRESH="${DIST_THRESH}" \
MAX_3D_DISTANCE="${MAX_3D_DISTANCE}" \
MAX_FRAMES="${MAX_FRAMES}" \
MERGE_OUTPUT_NAME="${MERGE_OUTPUT_NAME}" \
FILE_PATTERN="${FILE_PATTERN}" \
ENABLE_USE_3D="${ENABLE_USE_3D}" \
bash "${TRACKING_LAUNCHER}" "${target_path}"
return
fi
PYTHON_BIN="${PYTHON_BIN}" \
RESULTS_ROOT="${target_path}" \
TRACK_CLASSES="${TRACK_CLASSES}" \
IOU_THRESH="${IOU_THRESH}" \
MAX_AGE="${MAX_AGE}" \
MIN_HITS="${MIN_HITS}" \
DIST_THRESH="${DIST_THRESH}" \
MAX_3D_DISTANCE="${MAX_3D_DISTANCE}" \
MAX_FRAMES="${MAX_FRAMES}" \
MERGE_OUTPUT_NAME="${MERGE_OUTPUT_NAME}" \
FILE_PATTERN="${FILE_PATTERN}" \
ENABLE_USE_3D="${ENABLE_USE_3D}" \
bash "${TRACKING_LAUNCHER}" "${target_path}"
}
run_conversion_wrapper() {
local target_path="$1"
local convert_args=("${target_path}")
for issue_id in "${ISSUE_IDS[@]}"; do
convert_args+=(--issue-id "${issue_id}")
done
echo ""
echo "######################################################################"
echo "# Feishu issue-data protocol conversion"
echo "######################################################################"
echo "Target path : ${target_path}"
echo "Tracking JSON : ${CONVERT_TRACKING_JSON_NAME}"
echo "Output dir name : ${CONVERT_OUTPUT_DIR_NAME}"
if [[ -n "${CONVERT_CAM_ID}" ]]; then
echo "Camera ID override: ${CONVERT_CAM_ID}"
fi
PYTHON_BIN="${PYTHON_BIN}" \
RESULTS_ROOT="${RESULTS_ROOT}" \
OUTPUT_DIR_NAME="${CONVERT_OUTPUT_DIR_NAME}" \
MERGE_JSON_NAME="${CONVERT_TRACKING_JSON_NAME}" \
CAM_ID="${CONVERT_CAM_ID}" \
bash "${ISSUE_CONVERT_SCRIPT}" "${convert_args[@]}"
}
run_visualization_wrapper() {
local target_path="$1"
local visualize_args=("${target_path}")
for issue_id in "${ISSUE_IDS[@]}"; do
visualize_args+=(--issue-id "${issue_id}")
done
echo ""
echo "######################################################################"
echo "# Feishu issue-data raw-image visualization"
echo "######################################################################"
echo "Target path : ${target_path}"
echo "Download root : ${VIS_DOWNLOAD_ROOT}"
if [[ -n "${VIS_OUTPUT_ROOT}" ]]; then
echo "Output root : ${VIS_OUTPUT_ROOT}"
else
echo "Output dir name: ${VIS_OUTPUT_DIR_NAME}"
fi
PYTHON_BIN="${PYTHON_BIN}" \
RESULTS_ROOT="${RESULTS_ROOT}" \
DOWNLOAD_ROOT="${VIS_DOWNLOAD_ROOT}" \
OUTPUT_ROOT="${VIS_OUTPUT_ROOT}" \
OUTPUT_DIR_NAME="${VIS_OUTPUT_DIR_NAME}" \
TRACKING_JSON_NAME="${VIS_TRACKING_JSON_NAME}" \
VIS_MAX_FRAMES="${VIS_MAX_FRAMES}" \
VIS_CLASS_ID="${VIS_CLASS_ID}" \
VIS_TRACK_IDS="${VIS_TRACK_IDS}" \
VIS_SHOW_TRAJECTORY="${VIS_SHOW_TRAJECTORY}" \
bash "${ISSUE_VISUALIZE_SCRIPT}" "${visualize_args[@]}"
}
run_temporal_wrapper() {
local target_path="$1"
local temporal_args=("${target_path}")
for issue_id in "${ISSUE_IDS[@]}"; do
temporal_args+=(--issue-id "${issue_id}")
done
if [[ -n "${TEMPORAL_CLASS_ID}" ]]; then
temporal_args+=(--class-id "${TEMPORAL_CLASS_ID}")
fi
if [[ -n "${TEMPORAL_FRAME_ID_START}" ]]; then
temporal_args+=(--frame-id-start "${TEMPORAL_FRAME_ID_START}")
fi
if [[ -n "${TEMPORAL_FRAME_ID_END}" ]]; then
temporal_args+=(--frame-id-end "${TEMPORAL_FRAME_ID_END}")
fi
if [[ -n "${TEMPORAL_TRACK_IDS}" ]]; then
# shellcheck disable=SC2206
local temporal_track_id_arr=(${TEMPORAL_TRACK_IDS})
for track_id in "${temporal_track_id_arr[@]}"; do
temporal_args+=(--track-id "${track_id}")
done
fi
echo ""
echo "######################################################################"
echo "# Feishu issue-data temporal observation"
echo "######################################################################"
echo "Target path : ${target_path}"
if [[ -n "${TEMPORAL_CLASS_ID}" ]]; then
echo "Class ID : ${TEMPORAL_CLASS_ID}"
fi
if [[ -n "${TEMPORAL_TRACK_IDS}" ]]; then
echo "Track IDs : ${TEMPORAL_TRACK_IDS}"
fi
if [[ -n "${TEMPORAL_FRAME_ID_START}" || -n "${TEMPORAL_FRAME_ID_END}" ]]; then
echo "Frame ID Range : [${TEMPORAL_FRAME_ID_START:-"-inf"}, ${TEMPORAL_FRAME_ID_END:-"+inf"}]"
fi
echo "Heading Source : ${TEMPORAL_HEADING_SOURCE}"
PYTHON_BIN="${PYTHON_BIN}" \
RESULTS_ROOT="${RESULTS_ROOT}" \
OUTPUT_ROOT="${TEMPORAL_OUTPUT_ROOT}" \
OUTPUT_DIR_NAME="${TEMPORAL_OUTPUT_DIR_NAME}" \
TRACKING_JSON_NAME="${TEMPORAL_JSON_NAME}" \
TEMPORAL_MIN_LENGTH="${TEMPORAL_MIN_LENGTH}" \
TEMPORAL_CLASS_ID="${TEMPORAL_CLASS_ID}" \
TEMPORAL_TRACK_IDS="${TEMPORAL_TRACK_IDS}" \
TEMPORAL_FRAME_ID_START="${TEMPORAL_FRAME_ID_START}" \
TEMPORAL_FRAME_ID_END="${TEMPORAL_FRAME_ID_END}" \
TEMPORAL_PLOTS="${TEMPORAL_PLOTS}" \
TEMPORAL_EXPORT_SERIES="${TEMPORAL_EXPORT_SERIES}" \
TEMPORAL_FOCUS_TRACK_PLOTS="${TEMPORAL_FOCUS_TRACK_PLOTS}" \
TEMPORAL_PREFER_EGO="${TEMPORAL_PREFER_EGO}" \
TEMPORAL_X_AXIS="${TEMPORAL_X_AXIS}" \
TEMPORAL_HEADING_SOURCE="${TEMPORAL_HEADING_SOURCE}" \
bash "${ISSUE_TEMPORAL_SCRIPT}" "${temporal_args[@]}"
}
if [[ "${#ISSUE_IDS[@]}" -eq 0 ]]; then
run_tracking_wrapper "${TARGET_PATH}"
workflow_failed=0
if [[ "${ENABLE_VISUALIZE}" == "1" ]]; then
if ! run_visualization_wrapper "${TARGET_PATH}"; then
workflow_failed=1
fi
fi
if [[ "${ENABLE_TEMPORAL_ANALYSIS}" == "1" ]]; then
if ! run_temporal_wrapper "${TARGET_PATH}"; then
workflow_failed=1
fi
fi
if [[ "${ENABLE_CONVERT}" == "1" ]]; then
if ! run_conversion_wrapper "${TARGET_PATH}"; then
workflow_failed=1
fi
fi
[[ "${workflow_failed}" -eq 0 ]]
exit 0
fi
total_issues=0
success_issues=0
failed_issues=0
for issue_index in "${!ISSUE_IDS[@]}"; do
issue_id="${ISSUE_IDS[issue_index]}"
issue_root="${TARGET_PATH}/issue_${issue_id}"
((total_issues += 1))
echo ""
printf '[TRACKING][%d/%d] issue_%s\n' \
"$((issue_index + 1))" "${#ISSUE_IDS[@]}" "${issue_id}"
if [[ ! -d "${issue_root}" ]]; then
echo "[FAIL] issue_${issue_id}: not found under ${TARGET_PATH}" >&2
((failed_issues += 1))
continue
fi
if run_tracking_wrapper "${issue_root}"; then
((success_issues += 1))
else
((failed_issues += 1))
echo "[FAIL] issue_${issue_id}: tracking failed" >&2
fi
done
echo ""
printf '[DONE] issues=%d success=%d failed=%d\n' \
"${total_issues}" "${success_issues}" "${failed_issues}"
workflow_failed=0
if [[ "${ENABLE_VISUALIZE}" == "1" ]]; then
if [[ "${failed_issues}" -ne 0 ]]; then
echo "Skipping visualization because some tracking jobs failed" >&2
else
if ! run_visualization_wrapper "${TARGET_PATH}"; then
workflow_failed=1
fi
fi
fi
if [[ "${ENABLE_TEMPORAL_ANALYSIS}" == "1" ]]; then
if [[ "${failed_issues}" -ne 0 ]]; then
echo "Skipping temporal observation because some tracking jobs failed" >&2
else
if ! run_temporal_wrapper "${TARGET_PATH}"; then
workflow_failed=1
fi
fi
fi
if [[ "${ENABLE_CONVERT}" == "1" ]]; then
if [[ "${failed_issues}" -ne 0 ]]; then
echo "Skipping protocol conversion because some tracking jobs failed" >&2
else
if ! run_conversion_wrapper "${TARGET_PATH}"; then
workflow_failed=1
fi
fi
fi
[[ "${failed_issues}" -eq 0 && "${workflow_failed}" -eq 0 ]]

View File

@@ -0,0 +1,260 @@
#!/usr/bin/env bash
set -euo pipefail
SCRIPT_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
PROJECT_ROOT=$(cd "${SCRIPT_DIR}/../.." && pwd)
export PYTHONPATH="${PROJECT_ROOT}:${PYTHONPATH:-}"
RESULTS_ROOT=${RESULTS_ROOT:-/data1/dongying/Mono3d/G1Q3/feishu_project/inference_issue_data}
DOWNLOAD_ROOT=${DOWNLOAD_ROOT:-/data1/dongying/Mono3d/G1Q3/feishu_project/downloaded_issue_data}
OUTPUT_ROOT=${OUTPUT_ROOT:-}
OUTPUT_DIR_NAME=${OUTPUT_DIR_NAME:-tracking_vis_raw}
PYTHON_BIN=${PYTHON_BIN:-/deeplearning_team/ydong/dongying/miniconda/envs/dev/bin/python}
VISUALIZE_SCRIPT=${VISUALIZE_SCRIPT:-"${PROJECT_ROOT}/tools/temporal_analysis/visualize_tracking_boxes.py"}
TRACKING_JSON_NAME=${TRACKING_JSON_NAME:-merge.json}
VIS_MAX_FRAMES=${VIS_MAX_FRAMES:-}
VIS_CLASS_ID=${VIS_CLASS_ID:-}
VIS_TRACK_IDS=${VIS_TRACK_IDS:-}
VIS_SHOW_TRAJECTORY=${VIS_SHOW_TRAJECTORY:-0}
TARGET_PATH=""
ISSUE_IDS=()
while (($# > 0)); do
case "$1" in
--issue-id)
if (($# < 2)); then
echo "Error: --issue-id requires a value" >&2
exit 1
fi
ISSUE_IDS+=("$2")
shift 2
;;
--issue-id=*)
ISSUE_IDS+=("${1#*=}")
shift
;;
-*)
echo "Error: unsupported option: $1" >&2
exit 1
;;
*)
if [[ -n "${TARGET_PATH}" ]]; then
echo "Error: multiple target paths provided: ${TARGET_PATH} and $1" >&2
exit 1
fi
TARGET_PATH="$1"
shift
;;
esac
done
if [[ -z "${TARGET_PATH}" ]]; then
TARGET_PATH="${RESULTS_ROOT}"
fi
if [[ ! -e "${TARGET_PATH}" ]]; then
echo "Error: target path does not exist: ${TARGET_PATH}" >&2
exit 1
fi
resolve_abs_path() {
local target_path="$1"
if [[ -d "${target_path}" ]]; then
(
cd "${target_path}"
pwd -P
)
return 0
fi
local parent_dir
parent_dir=$(
cd "$(dirname "${target_path}")"
pwd -P
)
printf '%s/%s\n' "${parent_dir}" "$(basename "${target_path}")"
}
RESULTS_ROOT_ABS="$(resolve_abs_path "${RESULTS_ROOT}")"
DOWNLOAD_ROOT_ABS="$(resolve_abs_path "${DOWNLOAD_ROOT}")"
is_case_dir() {
local dir_path="$1"
[[ -d "${dir_path}" ]] && [[ -f "${dir_path}/${TRACKING_JSON_NAME}" ]]
}
derive_output_dir() {
local case_dir="$1"
local case_abs
local rel_case_dir
case_abs="$(resolve_abs_path "${case_dir}")"
if [[ -n "${OUTPUT_ROOT}" ]]; then
if [[ "${case_abs}" == "${RESULTS_ROOT_ABS}" ]]; then
printf '%s\n' "${OUTPUT_ROOT}"
return 0
fi
if [[ "${case_abs}" == "${RESULTS_ROOT_ABS}"/* ]]; then
rel_case_dir="${case_abs#"${RESULTS_ROOT_ABS}/"}"
printf '%s/%s\n' "${OUTPUT_ROOT%/}" "${rel_case_dir}"
return 0
fi
printf '%s/%s\n' "${OUTPUT_ROOT%/}" "$(basename "${case_abs}")"
return 0
fi
printf '%s/%s\n' "${case_abs}" "${OUTPUT_DIR_NAME}"
}
derive_raw_video_path() {
local case_dir="$1"
local case_abs
local rel_case_dir
case_abs="$(resolve_abs_path "${case_dir}")"
if [[ "${case_abs}" != "${RESULTS_ROOT_ABS}" && "${case_abs}" != "${RESULTS_ROOT_ABS}"/* ]]; then
echo "Error: case directory is not under RESULTS_ROOT: ${case_abs}" >&2
return 1
fi
if [[ "${case_abs}" == "${RESULTS_ROOT_ABS}" ]]; then
echo "Error: cannot derive a raw video path from the results root itself" >&2
return 1
fi
rel_case_dir="${case_abs#"${RESULTS_ROOT_ABS}/"}"
printf '%s/%s/sigmastar.1/camera4.bin\n' "${DOWNLOAD_ROOT_ABS}" "${rel_case_dir}"
}
run_single_case() {
local case_dir="$1"
local tracking_json="${case_dir}/${TRACKING_JSON_NAME}"
local raw_video_path
local output_dir
local cmd
if [[ ! -f "${tracking_json}" ]]; then
echo "[ERROR] ${TRACKING_JSON_NAME} not found in case directory: ${case_dir}" >&2
return 1
fi
raw_video_path="$(derive_raw_video_path "${case_dir}")"
if [[ ! -f "${raw_video_path}" ]]; then
echo "[ERROR] camera4.bin not found for case: ${case_dir}" >&2
echo " expected: ${raw_video_path}" >&2
return 1
fi
output_dir="$(derive_output_dir "${case_dir}")"
cmd=(
"${PYTHON_BIN}" "${VISUALIZE_SCRIPT}"
--tracking "${tracking_json}"
--images "${raw_video_path}"
--output "${output_dir}"
--align-by-frame-id
)
if [[ -n "${VIS_MAX_FRAMES}" ]]; then
cmd+=(--max-frames "${VIS_MAX_FRAMES}")
fi
if [[ -n "${VIS_CLASS_ID}" ]]; then
cmd+=(--class-id "${VIS_CLASS_ID}")
fi
if [[ -n "${VIS_TRACK_IDS}" ]]; then
# shellcheck disable=SC2206
local track_id_arr=(${VIS_TRACK_IDS})
cmd+=(--track-ids "${track_id_arr[@]}")
fi
if [[ "${VIS_SHOW_TRAJECTORY}" == "1" ]]; then
cmd+=(--show-trajectory)
fi
echo ""
echo "######################################################################"
echo "# Feishu issue-data raw-image visualization"
echo "######################################################################"
echo "Case : ${case_dir}"
echo "Track : ${tracking_json}"
echo "Video : ${raw_video_path}"
echo "Output : ${output_dir}"
"${cmd[@]}"
}
run_batch_root() {
local batch_root="$1"
local total_cases=0
local success_cases=0
local failed_cases=0
while IFS= read -r -d '' tracking_json; do
local case_dir
case_dir="$(dirname "${tracking_json}")"
((total_cases += 1))
if run_single_case "${case_dir}"; then
((success_cases += 1))
else
((failed_cases += 1))
printf '[FAIL] case=%s\n' "${case_dir}" >&2
fi
done < <(
find "${batch_root}" -type f -name "${TRACKING_JSON_NAME}" -print0 | sort -z
)
if [[ "${total_cases}" -eq 0 ]]; then
echo "[ERROR] No ${TRACKING_JSON_NAME} files were found under: ${batch_root}" >&2
return 1
fi
echo ""
printf '[DONE] cases=%d success=%d failed=%d\n' \
"${total_cases}" "${success_cases}" "${failed_cases}"
[[ "${failed_cases}" -eq 0 ]]
}
if [[ "${#ISSUE_IDS[@]}" -eq 0 ]]; then
if is_case_dir "${TARGET_PATH}"; then
run_single_case "${TARGET_PATH}"
elif [[ -d "${TARGET_PATH}" ]]; then
run_batch_root "${TARGET_PATH}"
else
echo "Error: unsupported target path: ${TARGET_PATH}" >&2
exit 1
fi
exit 0
fi
total_issues=0
success_issues=0
failed_issues=0
for issue_id in "${ISSUE_IDS[@]}"; do
issue_root="${TARGET_PATH}/issue_${issue_id}"
((total_issues += 1))
if [[ ! -d "${issue_root}" ]]; then
echo "[FAIL] issue_${issue_id}: not found under ${TARGET_PATH}" >&2
((failed_issues += 1))
continue
fi
if run_batch_root "${issue_root}"; then
((success_issues += 1))
else
((failed_issues += 1))
echo "[FAIL] issue_${issue_id}: visualization failed" >&2
fi
done
echo ""
printf '[DONE] issues=%d success=%d failed=%d\n' \
"${total_issues}" "${success_issues}" "${failed_issues}"
[[ "${failed_issues}" -eq 0 ]]

View File

@@ -0,0 +1,23 @@
#!/usr/bin/env bash
set -euo pipefail
SCRIPT_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
PROJECT_ROOT=$(cd "${SCRIPT_DIR}/../.." && pwd)
SYNC_ROOT=${SYNC_ROOT:-/data1/dongying/Mono3d/G1Q3/feishu_project}
INPUT_JSON=${INPUT_JSON:-"${SYNC_ROOT}/exports/dongying_g1q3_issue_list.json"}
OUTPUT_DIR=${OUTPUT_DIR:-"${SYNC_ROOT}/reports/issue_tag_profile"}
REPORT_NAME=${REPORT_NAME:-dongying_g1q3_issue_tag_profile}
PYTHON_BIN=${PYTHON_BIN:-/deeplearning_team/ydong/dongying/miniconda/envs/dev/bin/python}
TOP_LIMIT=${TOP_LIMIT:-15}
CMD=(
"${PYTHON_BIN}" "${PROJECT_ROOT}/tools/feishu_project/analyze_issue_tag_profile.py"
--input-json "${INPUT_JSON}"
--output-dir "${OUTPUT_DIR}"
--report-name "${REPORT_NAME}"
--top-limit "${TOP_LIMIT}"
)
CMD+=("$@")
"${CMD[@]}"

View File

@@ -0,0 +1,583 @@
#!/usr/bin/env python3
"""Run ROI1 crop-center compensation experiments and compare detection stability."""
from __future__ import annotations
import argparse
import json
import math
import re
import statistics
import subprocess
import sys
from pathlib import Path
from typing import Any
ROOT = Path(__file__).resolve().parents[2]
DEFAULT_MODEL = ROOT / "runs" / "export" / "train_mono3d_two_roi_20260416-raw_no_edge" / "merged_model.torchscript"
DEFAULT_OUTPUT_ROOT = Path("/data1/dongying/Mono3d/G1Q3/feishu_project/roi1_crop_compensation_experiments")
FRAME_FILE_RE = re.compile(r"camera4_(\d+)_")
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--baseline-case-dir", required=True, type=Path, help="Baseline inference case output directory.")
parser.add_argument("--video-case-dir", required=True, type=Path, help="Input video case path passed to --video-case-dir.")
parser.add_argument("--tracking-json", type=Path, default=None, help="Tracking JSON used to derive the reference target trajectory.")
parser.add_argument("--track-id", type=int, default=7)
parser.add_argument("--frame-id-start", type=int, required=True)
parser.add_argument("--frame-id-end", type=int, required=True)
parser.add_argument("--alpha", type=float, default=1.0, help="Scale factor applied to the bbox-derived ROI1 compensation.")
parser.add_argument("--exported-model", type=Path, default=DEFAULT_MODEL)
parser.add_argument("--device", type=str, default="cuda")
parser.add_argument("--output-root", type=Path, default=DEFAULT_OUTPUT_ROOT)
parser.add_argument("--skip-existing", action="store_true", help="Reuse existing experiment outputs when present.")
return parser.parse_args()
def load_json(path: Path) -> Any:
with path.open("r", encoding="utf-8") as file:
return json.load(file)
def save_json(path: Path, payload: Any) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
with path.open("w", encoding="utf-8") as file:
json.dump(payload, file, ensure_ascii=False, indent=2)
file.write("\n")
def coerce_bbox(values: Any) -> tuple[float, float, float, float] | None:
if not isinstance(values, (list, tuple)) or len(values) < 4:
return None
try:
x1, y1, x2, y2 = (float(values[0]), float(values[1]), float(values[2]), float(values[3]))
except (TypeError, ValueError):
return None
return x1, y1, x2, y2
def bbox_iou(box_a: tuple[float, float, float, float], box_b: tuple[float, float, float, float]) -> float:
ax1, ay1, ax2, ay2 = box_a
bx1, by1, bx2, by2 = box_b
inter_x1 = max(ax1, bx1)
inter_y1 = max(ay1, by1)
inter_x2 = min(ax2, bx2)
inter_y2 = min(ay2, by2)
inter_w = max(0.0, inter_x2 - inter_x1)
inter_h = max(0.0, inter_y2 - inter_y1)
inter_area = inter_w * inter_h
if inter_area <= 0.0:
return 0.0
area_a = max(0.0, ax2 - ax1) * max(0.0, ay2 - ay1)
area_b = max(0.0, bx2 - bx1) * max(0.0, by2 - by1)
denom = area_a + area_b - inter_area
return inter_area / denom if denom > 0.0 else 0.0
def center_distance(box_a: tuple[float, float, float, float], box_b: tuple[float, float, float, float]) -> float:
acx = (box_a[0] + box_a[2]) * 0.5
acy = (box_a[1] + box_a[3]) * 0.5
bcx = (box_b[0] + box_b[2]) * 0.5
bcy = (box_b[1] + box_b[3]) * 0.5
return math.hypot(acx - bcx, acy - bcy)
def frame_id_from_tracking_frame(frame_data: dict[str, Any], fallback_idx: int) -> int:
for det in frame_data.get("detections", []):
for key in ("frameId", "frame_id"):
value = det.get(key)
if value is None:
continue
try:
return int(value)
except (TypeError, ValueError):
continue
frame_info = frame_data.get("frame_info")
if isinstance(frame_info, dict):
for key in ("frame_id", "frameId", "original_frame_id"):
value = frame_info.get(key)
if value is None:
continue
try:
return int(value)
except (TypeError, ValueError):
continue
return fallback_idx
def load_reference_track(tracking_json: Path, track_id: int, frame_id_start: int, frame_id_end: int) -> list[dict[str, Any]]:
payload = load_json(tracking_json)
frames = payload.get("frames", payload) if isinstance(payload, dict) else payload
if not isinstance(frames, list):
raise ValueError(f"Unsupported tracking JSON structure in {tracking_json}")
reference = []
for frame_idx, frame_data in enumerate(frames):
if not isinstance(frame_data, dict):
continue
frame_id = frame_id_from_tracking_frame(frame_data, frame_idx)
if frame_id < frame_id_start or frame_id > frame_id_end:
continue
for det in frame_data.get("detections", []):
if det.get("track_id") != track_id:
continue
bbox = coerce_bbox(det.get("bbox"))
if bbox is None:
continue
class_id = int(det.get("class_id")) if det.get("class_id") is not None else None
reference.append(
{
"frame_id": frame_id,
"frame_idx": frame_idx,
"bbox": bbox,
"class_id": class_id,
"type_name": det.get("type_name"),
"y2": bbox[3],
}
)
break
reference.sort(key=lambda item: item["frame_id"])
if not reference:
raise FileNotFoundError(
f"Track {track_id} not found in {tracking_json} within frame_id [{frame_id_start}, {frame_id_end}]"
)
return reference
def build_offset_maps(
reference_track: list[dict[str, Any]],
alpha: float,
) -> tuple[dict[int, float], dict[int, float], dict[int, float]]:
ref_y2 = float(reference_track[0]["y2"])
oracle: dict[int, float] = {}
causal: dict[int, float] = {}
frame_delta: dict[int, float] = {}
prev_oracle_offset = 0.0
prev_y2 = ref_y2
for idx, item in enumerate(reference_track):
frame_id = int(item["frame_id"])
current_offset = alpha * (float(item["y2"]) - ref_y2)
oracle[frame_id] = current_offset
causal[frame_id] = prev_oracle_offset if idx > 0 else 0.0
frame_delta[frame_id] = alpha * (float(item["y2"]) - prev_y2) if idx > 0 else 0.0
prev_oracle_offset = current_offset
prev_y2 = float(item["y2"])
return oracle, causal, frame_delta
def write_offset_map(path: Path, offsets: dict[int, float], metadata: dict[str, Any]) -> None:
save_json(
path,
{
"default_offset_px": 0.0,
"frame_id_offsets": {str(frame_id): offset for frame_id, offset in offsets.items()},
"metadata": metadata,
},
)
def build_prediction_frame_map(predictions_merge_dir: Path) -> dict[int, Path]:
frame_map: dict[int, Path] = {}
for json_path in sorted(predictions_merge_dir.glob("*.json")):
match = FRAME_FILE_RE.search(json_path.name)
if not match:
continue
frame_map[int(match.group(1))] = json_path
return frame_map
def extract_candidate_records(frame_json_path: Path) -> list[dict[str, Any]]:
payload = load_json(frame_json_path)
if not isinstance(payload, dict):
raise ValueError(f"Unexpected frame prediction structure in {frame_json_path}")
candidates = []
for value in payload.values():
if not isinstance(value, dict):
continue
bbox = coerce_bbox(value.get("box2d"))
if bbox is None:
continue
class_id = value.get("type")
try:
class_id = int(class_id) if class_id is not None else None
except (TypeError, ValueError):
class_id = None
x_ego = None
if isinstance(value.get("box_center_xyz_ego"), list) and value["box_center_xyz_ego"]:
try:
x_ego = float(value["box_center_xyz_ego"][0])
except (TypeError, ValueError):
x_ego = None
if x_ego is None and isinstance(value.get("xyzlhwyaw_ego"), list) and value["xyzlhwyaw_ego"]:
try:
x_ego = float(value["xyzlhwyaw_ego"][0])
except (TypeError, ValueError):
x_ego = None
if x_ego is None:
continue
candidates.append(
{
"bbox": bbox,
"class_id": class_id,
"type_name": value.get("type_name"),
"score": float(value.get("score", 0.0)),
"x_ego": x_ego,
}
)
return candidates
def match_reference_detection(
frame_json_path: Path,
reference_bbox: tuple[float, float, float, float],
reference_class_id: int | None,
) -> dict[str, Any] | None:
candidates = extract_candidate_records(frame_json_path)
same_class = [
candidate for candidate in candidates if reference_class_id is not None and candidate["class_id"] == reference_class_id
]
candidate_pool = same_class or candidates
if not candidate_pool:
return None
best = max(
candidate_pool,
key=lambda candidate: (
bbox_iou(candidate["bbox"], reference_bbox),
-center_distance(candidate["bbox"], reference_bbox),
candidate["score"],
),
)
best = dict(best)
best["iou_to_reference"] = bbox_iou(best["bbox"], reference_bbox)
return best
def collect_variant_series(reference_track: list[dict[str, Any]], predictions_merge_dir: Path) -> list[dict[str, Any]]:
frame_map = build_prediction_frame_map(predictions_merge_dir)
series: list[dict[str, Any]] = []
for item in reference_track:
frame_id = int(item["frame_id"])
frame_json_path = frame_map.get(frame_id)
if frame_json_path is None:
continue
matched = match_reference_detection(frame_json_path, item["bbox"], item["class_id"])
if matched is None:
continue
bbox = matched["bbox"]
series.append(
{
"frame_id": frame_id,
"x_ego": float(matched["x_ego"]),
"score": float(matched["score"]),
"iou_to_reference": float(matched["iou_to_reference"]),
"bbox": bbox,
"y2": bbox[3],
"cy": (bbox[1] + bbox[3]) * 0.5,
"w": bbox[2] - bbox[0],
"h": bbox[3] - bbox[1],
}
)
series.sort(key=lambda item: item["frame_id"])
return series
def percentile(sorted_values: list[float], q: float) -> float:
if not sorted_values:
raise ValueError("sorted_values must not be empty")
if q <= 0:
return sorted_values[0]
if q >= 1:
return sorted_values[-1]
index = max(0, math.ceil(q * len(sorted_values)) - 1)
return sorted_values[index]
def compute_series_metrics(series: list[dict[str, Any]]) -> dict[str, Any]:
if len(series) < 2:
raise ValueError("Series must contain at least two samples")
x_values = [item["x_ego"] for item in series]
y2_values = [item["y2"] for item in series]
cy_values = [item["cy"] for item in series]
ious = [item["iou_to_reference"] for item in series]
scores = [item["score"] for item in series]
dx = [x_values[idx] - x_values[idx - 1] for idx in range(1, len(x_values))]
dy2 = [y2_values[idx] - y2_values[idx - 1] for idx in range(1, len(y2_values))]
dcy = [cy_values[idx] - cy_values[idx - 1] for idx in range(1, len(cy_values))]
abs_dx = sorted(abs(value) for value in dx)
abs_dy2 = sorted(abs(value) for value in dy2)
abs_dcy = sorted(abs(value) for value in dcy)
window = 5
local_dev = []
for idx in range(window, len(x_values) - window):
local_mean = sum(x_values[idx - window : idx + window + 1]) / (2 * window + 1)
local_dev.append(abs(x_values[idx] - local_mean))
local_dev_sorted = sorted(local_dev) if local_dev else [0.0]
return {
"samples": len(series),
"frame_id_start": int(series[0]["frame_id"]),
"frame_id_end": int(series[-1]["frame_id"]),
"x_start": float(x_values[0]),
"x_end": float(x_values[-1]),
"x_change": float(x_values[-1] - x_values[0]),
"abs_dx_mean": float(statistics.mean(abs(value) for value in dx)),
"abs_dx_p95": float(percentile(abs_dx, 0.95)),
"abs_dx_max": float(max(abs_dx)),
"abs_dy2_mean": float(statistics.mean(abs(value) for value in dy2)),
"abs_dy2_p95": float(percentile(abs_dy2, 0.95)),
"abs_dcy_mean": float(statistics.mean(abs(value) for value in dcy)),
"abs_dcy_p95": float(percentile(abs_dcy, 0.95)),
"local_dev_mean": float(statistics.mean(local_dev_sorted)),
"local_dev_p95": float(percentile(local_dev_sorted, 0.95)),
"local_dev_max": float(max(local_dev_sorted)),
"mean_iou_to_reference": float(statistics.mean(ious)),
"min_iou_to_reference": float(min(ious)),
"mean_score": float(statistics.mean(scores)),
}
def run_inference_variant(
*,
video_case_dir: Path,
output_dir: Path,
frame_id_start: int,
frame_id_end: int,
exported_model: Path,
device: str,
offset_map_path: Path | None = None,
) -> None:
cmd = [
sys.executable,
str(ROOT / "tools" / "model_inference" / "core" / "run_two_roi_exported_onnx_infer.py"),
"--video-case-dir",
str(video_case_dir),
"--output-dir",
str(output_dir),
"--frame-id-start",
str(frame_id_start),
"--frame-id-end",
str(frame_id_end),
"--video-stride",
"1",
"--exported-model",
str(exported_model),
"--device",
device,
"--enable-cross-class-merge-prior",
"--save-aggregate-predictions",
]
if offset_map_path is not None:
cmd.extend(["--roi1-crop-center-y-offset-map", str(offset_map_path)])
subprocess.run(cmd, check=True, cwd=str(ROOT))
def build_report_payload(
*,
args: argparse.Namespace,
reference_track: list[dict[str, Any]],
baseline_metrics: dict[str, Any],
oracle_metrics: dict[str, Any],
causal_metrics: dict[str, Any],
frame_delta_metrics: dict[str, Any],
oracle_offsets_path: Path,
causal_offsets_path: Path,
frame_delta_offsets_path: Path,
baseline_dir: Path,
oracle_dir: Path,
causal_dir: Path,
frame_delta_dir: Path,
) -> dict[str, Any]:
baseline_abs_dx_p95 = baseline_metrics["abs_dx_p95"]
baseline_local_dev_p95 = baseline_metrics["local_dev_p95"]
def compare_metrics(metrics: dict[str, Any]) -> dict[str, Any]:
return {
"abs_dx_p95_delta": float(metrics["abs_dx_p95"] - baseline_abs_dx_p95),
"abs_dx_p95_reduction_ratio": float((baseline_abs_dx_p95 - metrics["abs_dx_p95"]) / baseline_abs_dx_p95)
if baseline_abs_dx_p95 > 0
else 0.0,
"local_dev_p95_delta": float(metrics["local_dev_p95"] - baseline_local_dev_p95),
"local_dev_p95_reduction_ratio": float((baseline_local_dev_p95 - metrics["local_dev_p95"]) / baseline_local_dev_p95)
if baseline_local_dev_p95 > 0
else 0.0,
}
return {
"track_id": int(args.track_id),
"frame_id_start": int(args.frame_id_start),
"frame_id_end": int(args.frame_id_end),
"alpha": float(args.alpha),
"exported_model": str(args.exported_model.resolve()),
"baseline_case_dir": str(baseline_dir.resolve()),
"video_case_dir": str(args.video_case_dir.resolve()),
"reference_track_frames": len(reference_track),
"oracle_offset_map": str(oracle_offsets_path.resolve()),
"causal_offset_map": str(causal_offsets_path.resolve()),
"frame_delta_offset_map": str(frame_delta_offsets_path.resolve()),
"variants": {
"baseline": {
"output_dir": str(baseline_dir.resolve()),
"metrics": baseline_metrics,
},
"oracle": {
"output_dir": str(oracle_dir.resolve()),
"metrics": oracle_metrics,
"comparison_to_baseline": compare_metrics(oracle_metrics),
},
"causal": {
"output_dir": str(causal_dir.resolve()),
"metrics": causal_metrics,
"comparison_to_baseline": compare_metrics(causal_metrics),
},
"frame_delta": {
"output_dir": str(frame_delta_dir.resolve()),
"metrics": frame_delta_metrics,
"comparison_to_baseline": compare_metrics(frame_delta_metrics),
},
},
}
def main() -> int:
args = parse_args()
baseline_case_dir = args.baseline_case_dir.resolve()
tracking_json = args.tracking_json.resolve() if args.tracking_json else baseline_case_dir / "merge.json"
reference_track = load_reference_track(
tracking_json=tracking_json,
track_id=args.track_id,
frame_id_start=args.frame_id_start,
frame_id_end=args.frame_id_end,
)
oracle_offsets, causal_offsets, frame_delta_offsets = build_offset_maps(reference_track, alpha=args.alpha)
case_tag = f"{baseline_case_dir.name}_track{args.track_id}_f{args.frame_id_start}_{args.frame_id_end}_a{str(args.alpha).replace('.', 'p')}"
output_root = args.output_root.resolve() / case_tag
configs_dir = output_root / "configs"
oracle_offsets_path = configs_dir / "oracle_roi1_offsets.json"
causal_offsets_path = configs_dir / "causal_prev_frame_roi1_offsets.json"
frame_delta_offsets_path = configs_dir / "frame_delta_roi1_offsets.json"
oracle_dir = output_root / "oracle"
causal_dir = output_root / "causal"
frame_delta_dir = output_root / "frame_delta"
report_path = output_root / "experiment_summary.json"
write_offset_map(
oracle_offsets_path,
oracle_offsets,
{
"mode": "same_frame_oracle",
"track_id": args.track_id,
"alpha": args.alpha,
"reference_frame_id": reference_track[0]["frame_id"],
"reference_y2": reference_track[0]["y2"],
},
)
write_offset_map(
causal_offsets_path,
causal_offsets,
{
"mode": "previous_frame_causal",
"track_id": args.track_id,
"alpha": args.alpha,
"reference_frame_id": reference_track[0]["frame_id"],
"reference_y2": reference_track[0]["y2"],
},
)
write_offset_map(
frame_delta_offsets_path,
frame_delta_offsets,
{
"mode": "same_frame_prev_delta",
"track_id": args.track_id,
"alpha": args.alpha,
"reference_frame_id": reference_track[0]["frame_id"],
"reference_y2": reference_track[0]["y2"],
},
)
if not args.skip_existing or not (oracle_dir / "predictions" / "merge").is_dir():
run_inference_variant(
video_case_dir=args.video_case_dir.resolve(),
output_dir=oracle_dir,
frame_id_start=args.frame_id_start,
frame_id_end=args.frame_id_end,
exported_model=args.exported_model.resolve(),
device=args.device,
offset_map_path=oracle_offsets_path,
)
if not args.skip_existing or not (causal_dir / "predictions" / "merge").is_dir():
run_inference_variant(
video_case_dir=args.video_case_dir.resolve(),
output_dir=causal_dir,
frame_id_start=args.frame_id_start,
frame_id_end=args.frame_id_end,
exported_model=args.exported_model.resolve(),
device=args.device,
offset_map_path=causal_offsets_path,
)
if not args.skip_existing or not (frame_delta_dir / "predictions" / "merge").is_dir():
run_inference_variant(
video_case_dir=args.video_case_dir.resolve(),
output_dir=frame_delta_dir,
frame_id_start=args.frame_id_start,
frame_id_end=args.frame_id_end,
exported_model=args.exported_model.resolve(),
device=args.device,
offset_map_path=frame_delta_offsets_path,
)
baseline_series = collect_variant_series(reference_track, baseline_case_dir / "predictions" / "merge")
oracle_series = collect_variant_series(reference_track, oracle_dir / "predictions" / "merge")
causal_series = collect_variant_series(reference_track, causal_dir / "predictions" / "merge")
frame_delta_series = collect_variant_series(reference_track, frame_delta_dir / "predictions" / "merge")
baseline_metrics = compute_series_metrics(baseline_series)
oracle_metrics = compute_series_metrics(oracle_series)
causal_metrics = compute_series_metrics(causal_series)
frame_delta_metrics = compute_series_metrics(frame_delta_series)
report_payload = build_report_payload(
args=args,
reference_track=reference_track,
baseline_metrics=baseline_metrics,
oracle_metrics=oracle_metrics,
causal_metrics=causal_metrics,
frame_delta_metrics=frame_delta_metrics,
oracle_offsets_path=oracle_offsets_path,
causal_offsets_path=causal_offsets_path,
frame_delta_offsets_path=frame_delta_offsets_path,
baseline_dir=baseline_case_dir,
oracle_dir=oracle_dir,
causal_dir=causal_dir,
frame_delta_dir=frame_delta_dir,
)
save_json(report_path, report_payload)
print("")
print("ROI1 crop compensation experiment summary")
print(f"summary_json: {report_path}")
print(f"baseline abs_dx_p95 : {baseline_metrics['abs_dx_p95']:.4f} m/frame")
print(f"oracle abs_dx_p95 : {oracle_metrics['abs_dx_p95']:.4f} m/frame")
print(f"causal abs_dx_p95 : {causal_metrics['abs_dx_p95']:.4f} m/frame")
print(f"delta abs_dx_p95 : {frame_delta_metrics['abs_dx_p95']:.4f} m/frame")
print(f"baseline local_dev_p95 : {baseline_metrics['local_dev_p95']:.4f} m")
print(f"oracle local_dev_p95 : {oracle_metrics['local_dev_p95']:.4f} m")
print(f"causal local_dev_p95 : {causal_metrics['local_dev_p95']:.4f} m")
print(f"delta local_dev_p95 : {frame_delta_metrics['local_dev_p95']:.4f} m")
return 0
if __name__ == "__main__":
sys.exit(main())

View File

@@ -0,0 +1,219 @@
---
name: feishu-doc
description: Read, create, append, replace, and structurally edit Feishu Docx documents with the feishu_doc tool. Use when Codex needs to work with Feishu docs, cloud docs, docx links, document blocks, tables, images, or file attachments in Feishu/Lark documents.
---
# Feishu Doc
Use the `feishu_doc` tool for Feishu Docx operations.
If the current environment does not expose `feishu_doc`, tell the user that the required tool is unavailable and stop instead of inventing a workaround.
## Extract the doc token
Extract `doc_token` from the document URL.
Example:
- `https://xxx.feishu.cn/docx/ABC123def` -> `ABC123def`
## Start with the right workflow
Choose the lightest workflow that fits the request.
### Read a document
1. Call `{"action":"read","doc_token":"..."}` first.
2. Inspect `hint`, `block_types`, and summary fields in the response.
3. If the document contains structured content such as tables or images, call `{"action":"list_blocks","doc_token":"..."}`.
4. Use `get_block` only when a single block needs closer inspection.
### Replace or append text content
Use markdown-oriented actions for plain document content:
- `write`: replace the entire document
- `append`: append content to the end
Use markdown for headings, lists, code blocks, quotes, links, and images.
Do not rely on markdown tables. Use table actions instead.
### Perform block-level edits
Use these actions when the user wants targeted updates instead of full-document replacement:
- `list_blocks`
- `get_block`
- `update_block`
- `delete_block`
Prefer block-level edits when preserving surrounding content matters.
## Core actions
### Read
```json
{ "action": "read", "doc_token": "ABC123def" }
```
### Write the whole document
```json
{ "action": "write", "doc_token": "ABC123def", "content": "# Title\n\nMarkdown content..." }
```
### Append content
```json
{ "action": "append", "doc_token": "ABC123def", "content": "Additional content" }
```
### Create a document
Always pass the requesting user's `open_id` as `owner_open_id` when available so the user automatically gets access to the new document.
```json
{ "action": "create", "title": "New Document", "owner_open_id": "ou_xxx" }
```
With a folder:
```json
{
"action": "create",
"title": "New Document",
"folder_token": "fldcnXXX",
"owner_open_id": "ou_xxx"
}
```
### Inspect or edit blocks
```json
{ "action": "list_blocks", "doc_token": "ABC123def" }
```
```json
{ "action": "get_block", "doc_token": "ABC123def", "block_id": "doxcnXXX" }
```
```json
{
"action": "update_block",
"doc_token": "ABC123def",
"block_id": "doxcnXXX",
"content": "New text"
}
```
```json
{ "action": "delete_block", "doc_token": "ABC123def", "block_id": "doxcnXXX" }
```
## Tables
Use table actions for real Docx tables.
### Create an empty table
```json
{
"action": "create_table",
"doc_token": "ABC123def",
"row_size": 2,
"column_size": 2,
"column_width": [200, 200]
}
```
Optional: include `parent_block_id` to insert under a specific block.
### Fill table cells
```json
{
"action": "write_table_cells",
"doc_token": "ABC123def",
"table_block_id": "doxcnTABLE",
"values": [
["A1", "B1"],
["A2", "B2"]
]
}
```
### Create and fill a table in one step
```json
{
"action": "create_table_with_values",
"doc_token": "ABC123def",
"row_size": 2,
"column_size": 2,
"column_width": [200, 200],
"values": [
["A1", "B1"],
["A2", "B2"]
]
}
```
## Images and file attachments
### Upload an image
Use exactly one of `url` or `file_path`.
```json
{
"action": "upload_image",
"doc_token": "ABC123def",
"url": "https://example.com/image.png"
}
```
```json
{
"action": "upload_image",
"doc_token": "ABC123def",
"file_path": "/tmp/image.png",
"parent_block_id": "doxcnParent",
"index": 5
}
```
For small images, scale the asset before uploading so it displays at a useful size in the document.
### Upload a file attachment
Use exactly one of `url` or `file_path`.
```json
{
"action": "upload_file",
"doc_token": "ABC123def",
"url": "https://example.com/report.pdf"
}
```
```json
{
"action": "upload_file",
"doc_token": "ABC123def",
"file_path": "/tmp/report.pdf",
"filename": "Q1-report.pdf"
}
```
Optional: include `parent_block_id`.
## Operating rules
- Start with `read` for unfamiliar documents.
- Escalate to `list_blocks` when `read` indicates structured content.
- Use `write` only when replacing the full document is intended.
- Prefer block-level edits when the user asks for localized changes.
- Use table actions instead of markdown tables.
- Preserve the user's access by passing `owner_open_id` on document creation when that identity is available.

View File

@@ -0,0 +1,4 @@
interface:
display_name: "Feishu Doc"
short_description: "Read and edit Feishu Docx documents with the feishu_doc tool."
default_prompt: "Use the feishu_doc tool to read, create, append, and structurally edit Feishu Docx documents when the user asks about Feishu docs, cloud docs, or docx links."

View File

@@ -0,0 +1,214 @@
---
name: feishu-project-issue-data
description: Use when working in the yolo26-3d repository and the user wants to read Feishu Project issue views, export a view to structured JSON, classify issue data addresses, download issue data, repair affected standard-path downloads, validate case completeness, or run batch inference on downloaded issue data. Covers fp CLI plus tools/feishu_project/export_feishu_view_issues.py, download_issue_data.py/sh, and run_issue_data_inference.py/sh, including the 董颖-G1Q3 workflow.
---
# Feishu Project Issue Data
Use the repo dev env for Python commands:
```bash
/root/.codex/skills/use-dongying-dev-env/scripts/with-dev-env.sh python ...
```
or:
```bash
/deeplearning_team/ydong/dongying/miniconda/envs/dev/bin/python ...
```
Current repo defaults are documented in [../../feishu_project.md](../../feishu_project.md).
## When to use
- Read or verify a Feishu Project issue view with `fp`
- Export a view such as `董颖-G1Q3` to structured JSON
- Work with `问题数据地址` / `问题数据地址_PDCL`
- Analyze issue tag profiles across `目标` / `问题`
- Download issue data into the local workspace
- Repair historical bad copies that only copied `sigmastar.1`
- Check whether downloaded cases are inference-ready
- Batch-run exported-model inference on downloaded issue data
## Core workflow
### 1. Verify Feishu access and view contents
Use `fp` directly when the user wants current data.
```bash
fp view list -p <project_key> -u <user_key> -t issue --name "<view_name>"
fp workitem list -o json -p <project_key> -u <user_key> --view "<view_name>" --all
fp workitem get <issue_id> -o json -p <project_key> -u <user_key> -t issue
```
### 2. Export structured issue JSON
Use [../../export_feishu_view_issues.py](../../export_feishu_view_issues.py).
```bash
python ../../export_feishu_view_issues.py \
--project-key <project_key> \
--user-key <user_key> \
--view-name "<view_name>" \
--output ../../dongying_g1q3_issue_list.json
```
The export should include:
- `缺陷标签池`
- `问题数据地址`
- `问题数据地址_PDCL`
- `问题发生frameid`
### 3. Interpret data-address fields
Treat these as the same download class:
- pure `ADAS_...::...` clip references
- `mdi raw -r ...` commands
Use this rule:
- `ADAS_xxx::yyy` is equivalent to `mdi raw -r ADAS_xxx::yyy -s .`
Standard paths use these normalization rules:
- rewrite `hfs/project-G1M3` or `project-G1M3` to `G1M3` when needed
- if the path ends with `sigmastar.1`, copy the parent case dir
- if the path ends with `sigmastar.1/camera4.bin`, copy the case dir above it
- if the copied case has no local `test_data/calibs/camera4.json`, sync a shared parent `test_data` directory when present
### 4. Analyze issue tag profiles
Use [../../analyze_issue_tag_profile.py](../../analyze_issue_tag_profile.py) directly, or the G1Q3 wrapper
[../../run_issue_tag_profile.sh](../../run_issue_tag_profile.sh).
```bash
bash ../../run_issue_tag_profile.sh
```
Defaults:
- input JSON: `/data1/dongying/Mono3d/G1Q3/feishu_project/exports/dongying_g1q3_issue_list.json`
- report root: `/data1/dongying/Mono3d/G1Q3/feishu_project/reports/issue_tag_profile`
Output is a single HTML profile report with inline SVG donut charts and `场地问题` / `路测问题` toggle views covering:
- label completeness and multi-label quality
- `目标` / `问题` distributions
- `目标 x 问题` matrix
- function-domain x problem matrix
Publish the latest HTML to the static intranet site directory with
[../../publish_issue_tag_profile_site.sh](../../publish_issue_tag_profile_site.sh).
```bash
bash ../../publish_issue_tag_profile_site.sh
```
To show or serve through a fixed intranet address, pass `INTRANET_HOST` or
`PUBLIC_HOST`:
```bash
INTRANET_HOST=192.168.2.169 bash ../../publish_issue_tag_profile_site.sh
INTRANET_HOST=192.168.2.169 bash ../../serve_issue_tag_profile_site.sh restart
```
The address must either belong to the current host or be handled by a reverse
proxy/static server at that intranet machine.
Defaults:
- site root: `/data1/dongying/Mono3d/G1Q3/feishu_project/site`
- site index: `/data1/dongying/Mono3d/G1Q3/feishu_project/site/issue_tag_profile/index.html`
- example URL: `http://<intranet-host>:8088/issue_tag_profile/`
- Nginx example: [../../nginx_issue_tag_profile.conf.example](../../nginx_issue_tag_profile.conf.example)
If Nginx is not available yet, start a lightweight static server with
[../../serve_issue_tag_profile_site.sh](../../serve_issue_tag_profile_site.sh):
```bash
bash ../../serve_issue_tag_profile_site.sh start
```
### 5. Download or repair issue data
Use [../../download_issue_data.sh](../../download_issue_data.sh) or [../../download_issue_data.py](../../download_issue_data.py).
Common modes:
```bash
DRY_RUN=1 bash ../../download_issue_data.sh
```
```bash
ONLY_REDOWNLOAD_AFFECTED_CASES=1 bash ../../download_issue_data.sh
```
```bash
SKIP_MDI=1 bash ../../download_issue_data.sh --issue-id <id>
```
Defaults:
- download root: `/data1/dongying/Mono3d/G1Q3/feishu_project/downloaded_issue_data`
- manifest: `<download_root>/download_manifest.json`
Use `ONLY_REDOWNLOAD_AFFECTED_CASES=1` to repair old standard-path copies that previously kept only `sigmastar.1`.
### 6. Validate inference readiness
Use the same case-resolution rules as [../../../model_inference/adapters/video_dir_inference_utils.py](../../../model_inference/adapters/video_dir_inference_utils.py).
A valid case must resolve:
- `*/sigmastar.1/camera4.bin`
- a reachable `camera4.json` from one of:
- `case_dir/test_data/calibs/camera4.json`
- `case_dir.parent/test_data/calibs/camera4.json`
- `case_dir/sigmastar.1/calibs/camera4.json`
- `case_dir/calibs/camera4.json`
Prefer validating with the actual inference path-resolution logic instead of ad hoc file checks.
### 7. Run batch inference on downloaded issue data
Use [../../run_issue_data_inference.sh](../../run_issue_data_inference.sh) or [../../run_issue_data_inference.py](../../run_issue_data_inference.py).
```bash
DRY_RUN=1 bash ../../run_issue_data_inference.sh
```
```bash
bash ../../run_issue_data_inference.sh
```
Behavior:
- recursively scans the download root for `*/sigmastar.1/camera4.bin`
- calls [../../../model_inference/core/run_two_roi_exported_onnx_infer.py](../../../model_inference/core/run_two_roi_exported_onnx_infer.py) with `--video-case-dir`
- mirrors the download-tree relative layout into the inference output root
Defaults:
- inference root: `/data1/dongying/Mono3d/G1Q3/feishu_project/inference_issue_data`
- manifest: `<inference_root>/inference_manifest.json`
Useful flags:
- `SKIP_EXISTING=1`
- `ENABLE_ATTR=1`
- `SAVE_AGGREGATE_PREDICTIONS=1`
- `VIDEO_STRIDE=<n>`
- `MAX_IMAGES=<n>`
## Current repo artifacts
These files are useful outputs, but they are not the source of truth for latest Feishu data:
- [../../dongying_g1q3_issue_list.json](../../dongying_g1q3_issue_list.json)
- [../../dongying_g1q3_data_address_summary.md](../../dongying_g1q3_data_address_summary.md)
- [../../dongying_g1q3_data_address_catalog.md](../../dongying_g1q3_data_address_catalog.md)
If the user asks for latest status, re-query Feishu with `fp` and regenerate outputs instead of trusting stale local exports.

View File

@@ -0,0 +1,3 @@
python3 -m http.server 8088 \
--bind 0.0.0.0 \
--directory /data1/dongying/Mono3d/G1Q3/feishu_project/site

View File

@@ -0,0 +1,164 @@
#!/usr/bin/env bash
set -euo pipefail
SCRIPT_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
PROJECT_ROOT=$(cd "${SCRIPT_DIR}/../.." && pwd)
PROJECT_KEY=${PROJECT_KEY:-68ef617fb371dc80a10641f7}
USER_KEY=${USER_KEY:-7550145433285312514}
VIEW_NAME=${VIEW_NAME:-董颖-G1Q3}
WORK_ITEM_TYPE=${WORK_ITEM_TYPE:-issue}
SYNC_ROOT=${SYNC_ROOT:-/data1/dongying/Mono3d/G1Q3/feishu_project}
SYNC_MANIFEST_PATH_WAS_SET=${SYNC_MANIFEST_PATH+x}
INFERENCE_ROOT_WAS_SET=${INFERENCE_ROOT+x}
INFERENCE_MANIFEST_PATH_WAS_SET=${INFERENCE_MANIFEST_PATH+x}
EXPORT_JSON=${EXPORT_JSON:-"${SYNC_ROOT}/exports/dongying_g1q3_issue_list.json"}
SYNC_MANIFEST_PATH=${SYNC_MANIFEST_PATH:-"${SYNC_ROOT}/exports/dongying_g1q3_issue_list.sync_manifest.json"}
SNAPSHOT_DIR=${SNAPSHOT_DIR:-"${SYNC_ROOT}/exports/history"}
DOWNLOAD_ROOT=${DOWNLOAD_ROOT:-"${SYNC_ROOT}/downloaded_issue_data"}
DOWNLOAD_MANIFEST_PATH=${DOWNLOAD_MANIFEST_PATH:-"${DOWNLOAD_ROOT}/download_manifest.json"}
INFERENCE_ROOT=${INFERENCE_ROOT:-"${SYNC_ROOT}/inference_issue_data"}
INFERENCE_MANIFEST_PATH=${INFERENCE_MANIFEST_PATH:-"${INFERENCE_ROOT}/inference_manifest.json"}
PYTHON_BIN=${PYTHON_BIN:-/deeplearning_team/ydong/dongying/miniconda/envs/dev/bin/python}
RUN_DOWNLOAD=${RUN_DOWNLOAD:-1}
RUN_INFERENCE=${RUN_INFERENCE:-1}
ENABLE_TRACKING=${ENABLE_TRACKING:-1}
ENABLE_CONVERT=${ENABLE_CONVERT:-1}
CONVERT_OUTPUT_DIR_NAME=${CONVERT_OUTPUT_DIR_NAME:-objectlist}
CONVERT_TRACKING_JSON_NAME=${CONVERT_TRACKING_JSON_NAME:-merge.json}
CONVERT_CAM_ID=${CONVERT_CAM_ID:-}
SAVE_SNAPSHOT=${SAVE_SNAPSHOT:-1}
REFRESH_CHANGED_ISSUES=${REFRESH_CHANGED_ISSUES:-0}
SKIP_EXISTING_INFERENCE=${SKIP_EXISTING_INFERENCE:-1}
DRY_RUN=${DRY_RUN:-0}
USE_ISSUE_FRAME_WINDOW=${USE_ISSUE_FRAME_WINDOW:-1}
FRAME_BEFORE=${FRAME_BEFORE:-200}
FRAME_AFTER=${FRAME_AFTER:-200}
MISSING_ISSUE_FRAME_POLICY=${MISSING_ISSUE_FRAME_POLICY:-full}
ISSUE_ID_MIN=${ISSUE_ID_MIN:-}
ISSUE_ID_MAX=${ISSUE_ID_MAX:-6979030662}
EXPORTED_MODEL=${EXPORTED_MODEL:-${PROJECT_ROOT}/runs/export/train_mono3d_two_roi_20260427-raw_no_edge/merged_model.torchscript}
# EXPORTED_MODEL=${EXPORTED_MODEL:-${PROJECT_ROOT}/runs/export/train_mono3d_two_roi_20260506-keep_fake_3d_branch/merged_model.torchscript}
ISSUE_TRACKING_SCRIPT=${ISSUE_TRACKING_SCRIPT:-"${PROJECT_ROOT}/tools/feishu_project/run_issue_data_tracking.sh"}
TRACKING_MODEL_VERSION=${TRACKING_MODEL_VERSION:-20260427}
ENABLE_RETEST=${ENABLE_RETEST:-1}
RETEST_NAME_KEYWORDS=${RETEST_NAME_KEYWORDS:-AEB,FCW,场地} # AEB,FCW,场地 ACC,HMI,LCC,TSR
RETEST_RUN_NAME=${RETEST_RUN_NAME:-$(date +%Y%m%d_%H%M%S)}
RETEST_ROOT=${RETEST_ROOT:-"${SYNC_ROOT}/retest_runs/${TRACKING_MODEL_VERSION}_${RETEST_RUN_NAME}"}
SHOW_DISTANCE_LABEL=${SHOW_DISTANCE_LABEL:-1}
DISTANCE_LABEL_MODE=${DISTANCE_LABEL_MODE:-depth}
DISTANCE_LABEL_PANELS=${DISTANCE_LABEL_PANELS:-3d}
export ENABLE_CONVERT
export CONVERT_OUTPUT_DIR_NAME
export CONVERT_TRACKING_JSON_NAME
export CONVERT_CAM_ID
if [[ "${ENABLE_RETEST}" == "1" ]]; then
if [[ -z "${INFERENCE_ROOT_WAS_SET}" ]]; then
INFERENCE_ROOT="${RETEST_ROOT}/inference_issue_data"
fi
if [[ -z "${INFERENCE_MANIFEST_PATH_WAS_SET}" ]]; then
INFERENCE_MANIFEST_PATH="${INFERENCE_ROOT}/inference_manifest.json"
fi
if [[ -z "${SYNC_MANIFEST_PATH_WAS_SET}" ]]; then
SYNC_MANIFEST_PATH="${RETEST_ROOT}/dongying_g1q3_issue_list.sync_manifest.json"
fi
fi
CMD=(
"${PYTHON_BIN}" "${PROJECT_ROOT}/tools/feishu_project/sync_issue_data.py"
--project-key "${PROJECT_KEY}"
--user-key "${USER_KEY}"
--view-name "${VIEW_NAME}"
--work-item-type "${WORK_ITEM_TYPE}"
--output-json "${EXPORT_JSON}"
--sync-manifest-path "${SYNC_MANIFEST_PATH}"
--snapshot-dir "${SNAPSHOT_DIR}"
--python-bin "${PYTHON_BIN}"
--download-root "${DOWNLOAD_ROOT}"
--download-manifest-path "${DOWNLOAD_MANIFEST_PATH}"
--inference-root "${INFERENCE_ROOT}"
--inference-manifest-path "${INFERENCE_MANIFEST_PATH}"
)
if [[ "${RUN_DOWNLOAD}" == "1" ]]; then
CMD+=(--run-download)
fi
if [[ "${RUN_INFERENCE}" == "1" ]]; then
CMD+=(--run-inference)
fi
if [[ "${USE_ISSUE_FRAME_WINDOW}" == "1" ]]; then
CMD+=(--use-issue-frame-window --frame-before "${FRAME_BEFORE}" --frame-after "${FRAME_AFTER}" --missing-issue-frame-policy "${MISSING_ISSUE_FRAME_POLICY}")
fi
if [[ "${ENABLE_TRACKING}" == "1" ]]; then
CMD+=(--run-tracking --issue-tracking-script "${ISSUE_TRACKING_SCRIPT}")
fi
if [[ "${SAVE_SNAPSHOT}" == "1" ]]; then
CMD+=(--save-snapshot)
fi
if [[ "${REFRESH_CHANGED_ISSUES}" == "1" ]]; then
CMD+=(--refresh-changed-issues)
fi
if [[ "${SKIP_EXISTING_INFERENCE}" == "1" ]]; then
CMD+=(--skip-existing-inference)
fi
if [[ "${DRY_RUN}" == "1" ]]; then
CMD+=(--dry-run)
fi
if [[ -n "${ISSUE_ID_MIN}" ]]; then
CMD+=(--issue-id-min "${ISSUE_ID_MIN}")
fi
if [[ -n "${ISSUE_ID_MAX}" ]]; then
CMD+=(--issue-id-max "${ISSUE_ID_MAX}")
fi
if [[ -n "${EXPORTED_MODEL}" ]]; then
CMD+=(--exported-model "${EXPORTED_MODEL}")
fi
if [[ -n "${TRACKING_MODEL_VERSION}" ]]; then
CMD+=(--tracking-model-version "${TRACKING_MODEL_VERSION}")
fi
if [[ "${SHOW_DISTANCE_LABEL}" == "1" ]]; then
CMD+=("--inference-arg=--show-distance-label")
if [[ -n "${DISTANCE_LABEL_MODE}" ]]; then
CMD+=("--inference-arg=--distance-label-mode" "--inference-arg=${DISTANCE_LABEL_MODE}")
fi
if [[ -n "${DISTANCE_LABEL_PANELS}" ]]; then
# shellcheck disable=SC2206
DISTANCE_LABEL_PANELS_ARR=(${DISTANCE_LABEL_PANELS})
CMD+=("--inference-arg=--distance-label-panels")
for panel in "${DISTANCE_LABEL_PANELS_ARR[@]}"; do
CMD+=("--inference-arg=${panel}")
done
fi
fi
if [[ "${ENABLE_RETEST}" == "1" ]] && [[ -n "${RETEST_NAME_KEYWORDS}" ]]; then
IFS=',' read -r -a RETEST_KEYWORD_ARRAY <<< "${RETEST_NAME_KEYWORDS}"
for keyword in "${RETEST_KEYWORD_ARRAY[@]}"; do
keyword="${keyword#"${keyword%%[![:space:]]*}"}"
keyword="${keyword%"${keyword##*[![:space:]]}"}"
if [[ -n "${keyword}" ]]; then
CMD+=(--issue-name-keyword "${keyword}")
fi
done
fi
CMD+=("$@")
"${CMD[@]}"

File diff suppressed because it is too large Load Diff