单目3D初始代码
This commit is contained in:
3
tools/feishu_project/.gitignore
vendored
Executable file
3
tools/feishu_project/.gitignore
vendored
Executable file
@@ -0,0 +1,3 @@
|
||||
__pycache__/
|
||||
*.pyc
|
||||
reports/
|
||||
160
tools/feishu_project/SKILL_fp.md
Executable file
160
tools/feishu_project/SKILL_fp.md
Executable file
@@ -0,0 +1,160 @@
|
||||
---
|
||||
name: feishu-project-issue-data
|
||||
description: Use when working in the yolo26-3d repository and the user wants to read Feishu Project issue views, export a view to structured JSON, classify issue data addresses, download issue data, repair affected standard-path downloads, validate case completeness, or run batch inference on downloaded issue data. Covers fp CLI plus tools/feishu_project/export_feishu_view_issues.py, download_issue_data.py/sh, and run_issue_data_inference.py/sh, including the 董颖-G1Q3 workflow.
|
||||
---
|
||||
|
||||
# Feishu Project Issue Data
|
||||
|
||||
Use the repo dev env for Python commands:
|
||||
|
||||
```bash
|
||||
/root/.codex/skills/use-dongying-dev-env/scripts/with-dev-env.sh python ...
|
||||
```
|
||||
|
||||
or:
|
||||
|
||||
```bash
|
||||
/deeplearning_team/ydong/dongying/miniconda/envs/dev/bin/python ...
|
||||
```
|
||||
|
||||
Current repo defaults are documented in [../../feishu_project.md](../../feishu_project.md).
|
||||
|
||||
## When to use
|
||||
|
||||
- Read or verify a Feishu Project issue view with `fp`
|
||||
- Export a view such as `董颖-G1Q3` to structured JSON
|
||||
- Work with `问题数据地址` / `问题数据地址_PDCL`
|
||||
- Download issue data into the local workspace
|
||||
- Repair historical bad copies that only copied `sigmastar.1`
|
||||
- Check whether downloaded cases are inference-ready
|
||||
- Batch-run exported-model inference on downloaded issue data
|
||||
|
||||
## Core workflow
|
||||
|
||||
### 1. Verify Feishu access and view contents
|
||||
|
||||
Use `fp` directly when the user wants current data.
|
||||
|
||||
```bash
|
||||
fp view list -p <project_key> -u <user_key> -t issue --name "<view_name>"
|
||||
fp workitem list -o json -p <project_key> -u <user_key> --view "<view_name>" --all
|
||||
fp workitem get <issue_id> -o json -p <project_key> -u <user_key> -t issue
|
||||
```
|
||||
|
||||
### 2. Export structured issue JSON
|
||||
|
||||
Use [../../export_feishu_view_issues.py](../../export_feishu_view_issues.py).
|
||||
|
||||
```bash
|
||||
python ../../export_feishu_view_issues.py \
|
||||
--project-key <project_key> \
|
||||
--user-key <user_key> \
|
||||
--view-name "<view_name>" \
|
||||
--output ../../dongying_g1q3_issue_list.json
|
||||
```
|
||||
|
||||
The export should include:
|
||||
|
||||
- `缺陷标签池`
|
||||
- `问题数据地址`
|
||||
- `问题数据地址_PDCL`
|
||||
- `问题发生frameid`
|
||||
|
||||
### 3. Interpret data-address fields
|
||||
|
||||
Treat these as the same download class:
|
||||
|
||||
- pure `ADAS_...::...` clip references
|
||||
- `mdi raw -r ...` commands
|
||||
|
||||
Use this rule:
|
||||
|
||||
- `ADAS_xxx::yyy` is equivalent to `mdi raw -r ADAS_xxx::yyy -s .`
|
||||
|
||||
Standard paths use these normalization rules:
|
||||
|
||||
- rewrite `hfs/project-G1M3` or `project-G1M3` to `G1M3` when needed
|
||||
- if the path ends with `sigmastar.1`, copy the parent case dir
|
||||
- if the path ends with `sigmastar.1/camera4.bin`, copy the case dir above it
|
||||
- if the copied case has no local `test_data/calibs/camera4.json`, sync a shared parent `test_data` directory when present
|
||||
|
||||
### 4. Download or repair issue data
|
||||
|
||||
Use [../../download_issue_data.sh](../../download_issue_data.sh) or [../../download_issue_data.py](../../download_issue_data.py).
|
||||
|
||||
Common modes:
|
||||
|
||||
```bash
|
||||
DRY_RUN=1 bash ../../download_issue_data.sh
|
||||
```
|
||||
|
||||
```bash
|
||||
ONLY_REDOWNLOAD_AFFECTED_CASES=1 bash ../../download_issue_data.sh
|
||||
```
|
||||
|
||||
```bash
|
||||
SKIP_MDI=1 bash ../../download_issue_data.sh --issue-id <id>
|
||||
```
|
||||
|
||||
Defaults:
|
||||
|
||||
- download root: `/data1/dongying/Mono3d/G1Q3/feishu_project/downloaded_issue_data`
|
||||
- manifest: `<download_root>/download_manifest.json`
|
||||
|
||||
Use `ONLY_REDOWNLOAD_AFFECTED_CASES=1` to repair old standard-path copies that previously kept only `sigmastar.1`.
|
||||
|
||||
### 5. Validate inference readiness
|
||||
|
||||
Use the same case-resolution rules as [../../../model_inference/adapters/video_dir_inference_utils.py](../../../model_inference/adapters/video_dir_inference_utils.py).
|
||||
|
||||
A valid case must resolve:
|
||||
|
||||
- `*/sigmastar.1/camera4.bin`
|
||||
- a reachable `camera4.json` from one of:
|
||||
- `case_dir/test_data/calibs/camera4.json`
|
||||
- `case_dir.parent/test_data/calibs/camera4.json`
|
||||
- `case_dir/sigmastar.1/calibs/camera4.json`
|
||||
- `case_dir/calibs/camera4.json`
|
||||
|
||||
Prefer validating with the actual inference path-resolution logic instead of ad hoc file checks.
|
||||
|
||||
### 6. Run batch inference on downloaded issue data
|
||||
|
||||
Use [../../run_issue_data_inference.sh](../../run_issue_data_inference.sh) or [../../run_issue_data_inference.py](../../run_issue_data_inference.py).
|
||||
|
||||
```bash
|
||||
DRY_RUN=1 bash ../../run_issue_data_inference.sh
|
||||
```
|
||||
|
||||
```bash
|
||||
bash ../../run_issue_data_inference.sh
|
||||
```
|
||||
|
||||
Behavior:
|
||||
|
||||
- recursively scans the download root for `*/sigmastar.1/camera4.bin`
|
||||
- calls [../../../model_inference/core/run_two_roi_exported_onnx_infer.py](../../../model_inference/core/run_two_roi_exported_onnx_infer.py) with `--video-case-dir`
|
||||
- mirrors the download-tree relative layout into the inference output root
|
||||
|
||||
Defaults:
|
||||
|
||||
- inference root: `/data1/dongying/Mono3d/G1Q3/feishu_project/inference_issue_data`
|
||||
- manifest: `<inference_root>/inference_manifest.json`
|
||||
|
||||
Useful flags:
|
||||
|
||||
- `SKIP_EXISTING=1`
|
||||
- `ENABLE_ATTR=1`
|
||||
- `SAVE_AGGREGATE_PREDICTIONS=1`
|
||||
- `VIDEO_STRIDE=<n>`
|
||||
- `MAX_IMAGES=<n>`
|
||||
|
||||
## Current repo artifacts
|
||||
|
||||
These files are useful outputs, but they are not the source of truth for latest Feishu data:
|
||||
|
||||
- [../../dongying_g1q3_issue_list.json](../../dongying_g1q3_issue_list.json)
|
||||
- [../../dongying_g1q3_data_address_summary.md](../../dongying_g1q3_data_address_summary.md)
|
||||
- [../../dongying_g1q3_data_address_catalog.md](../../dongying_g1q3_data_address_catalog.md)
|
||||
|
||||
If the user asks for latest status, re-query Feishu with `fp` and regenerate outputs instead of trusting stale local exports.
|
||||
1274
tools/feishu_project/analyze_issue_tag_profile.py
Executable file
1274
tools/feishu_project/analyze_issue_tag_profile.py
Executable file
File diff suppressed because it is too large
Load Diff
221
tools/feishu_project/build_cncap_case_issue_json.py
Executable file
221
tools/feishu_project/build_cncap_case_issue_json.py
Executable file
@@ -0,0 +1,221 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Build a synthetic issue JSON from a plain CNCAP case list."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
from collections import Counter
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
|
||||
DEFAULT_INPUT = Path(__file__).with_name("cncap_case.txt")
|
||||
DEFAULT_ID_BASE = 9_000_000_000
|
||||
PDCL_REF_RE = re.compile(r"ADAS_[^:/\\\s]+::[^/\\\s]*")
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument(
|
||||
"--input",
|
||||
type=Path,
|
||||
default=DEFAULT_INPUT,
|
||||
help="Path to the CNCAP case list text file.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--output",
|
||||
type=Path,
|
||||
required=True,
|
||||
help="Path to the synthetic issue JSON output.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--case-index-output",
|
||||
type=Path,
|
||||
default=None,
|
||||
help="Optional companion index JSON output path.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--id-base",
|
||||
type=int,
|
||||
default=DEFAULT_ID_BASE,
|
||||
help="Synthetic issue id base. The first generated issue id is id_base + 1.",
|
||||
)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def parse_case_line(raw_line: str, line_no: int) -> tuple[str, str] | None:
|
||||
stripped = raw_line.strip()
|
||||
if not stripped or stripped.startswith("#"):
|
||||
return None
|
||||
|
||||
if "|" in stripped:
|
||||
case_name, source = (part.strip() for part in stripped.split("|", 1))
|
||||
else:
|
||||
try:
|
||||
case_name, source = stripped.rsplit(None, 1)
|
||||
except ValueError as exc:
|
||||
raise ValueError(
|
||||
f"Line {line_no}: expected '<case_name> <case_ref>' or '<case_name> | <case_ref>', "
|
||||
f"got: {raw_line.rstrip()!r}"
|
||||
) from exc
|
||||
|
||||
if not case_name or not source:
|
||||
raise ValueError(f"Line {line_no}: empty case name or case reference")
|
||||
return case_name, source
|
||||
|
||||
|
||||
def classify_source(source: str, line_no: int) -> tuple[str, str, str]:
|
||||
if "mdi raw" in source.lower() or PDCL_REF_RE.search(source):
|
||||
return "", source, "pdcl_mdi_download"
|
||||
if "/" in source or "\\" in source:
|
||||
return source, "", "standard_path"
|
||||
raise ValueError(f"Line {line_no}: unsupported case reference format: {source!r}")
|
||||
|
||||
|
||||
def build_payload(
|
||||
*,
|
||||
input_path: Path,
|
||||
case_rows: list[dict[str, Any]],
|
||||
id_base: int,
|
||||
) -> tuple[dict[str, Any], dict[str, Any]]:
|
||||
exported_at = datetime.now().astimezone().isoformat(timespec="seconds")
|
||||
name_counter: Counter[str] = Counter()
|
||||
items: list[dict[str, Any]] = []
|
||||
index_cases: list[dict[str, Any]] = []
|
||||
source_kind_counts: Counter[str] = Counter()
|
||||
|
||||
for ordinal, row in enumerate(case_rows, start=1):
|
||||
case_name = row["case_name"]
|
||||
source = row["source"]
|
||||
line_no = row["line_no"]
|
||||
raw_line = row["raw_line"]
|
||||
name_counter[case_name] += 1
|
||||
occurrence_index = name_counter[case_name]
|
||||
issue_id = id_base + ordinal
|
||||
|
||||
standard_path, pdcl_ref, source_kind = classify_source(source, line_no)
|
||||
source_kind_counts[source_kind] += 1
|
||||
|
||||
item = {
|
||||
"id": issue_id,
|
||||
"name": case_name,
|
||||
"status": "CNCAP_CASE",
|
||||
"created_at": None,
|
||||
"updated_at": exported_at,
|
||||
"问题数据地址": standard_path,
|
||||
"问题数据地址_PDCL": pdcl_ref,
|
||||
"问题发生frameid": None,
|
||||
"source_line_no": line_no,
|
||||
"case_ref": source,
|
||||
"case_occurrence_index": occurrence_index,
|
||||
"synthetic_issue_key": f"cncap_case_{ordinal:03d}",
|
||||
}
|
||||
items.append(item)
|
||||
index_cases.append(
|
||||
{
|
||||
"id": issue_id,
|
||||
"name": case_name,
|
||||
"source_kind": source_kind,
|
||||
"source_line_no": line_no,
|
||||
"case_ref": source,
|
||||
"case_occurrence_index": occurrence_index,
|
||||
"raw_line": raw_line,
|
||||
}
|
||||
)
|
||||
|
||||
duplicate_names = sorted(name for name, count in name_counter.items() if count > 1)
|
||||
payload = {
|
||||
"exported_at": exported_at,
|
||||
"source": {
|
||||
"type": "cncap_case_txt",
|
||||
"input_path": str(input_path.resolve()),
|
||||
"id_base": id_base,
|
||||
"work_item_type": "synthetic_issue",
|
||||
"included_detail_fields": [
|
||||
"问题数据地址",
|
||||
"问题数据地址_PDCL",
|
||||
"问题发生frameid",
|
||||
],
|
||||
},
|
||||
"summary": {
|
||||
"success": True,
|
||||
"total": len(items),
|
||||
"duplicate_name_count": len(duplicate_names),
|
||||
"pdcl_case_count": source_kind_counts.get("pdcl_mdi_download", 0),
|
||||
"standard_path_case_count": source_kind_counts.get("standard_path", 0),
|
||||
},
|
||||
"items": items,
|
||||
}
|
||||
case_index = {
|
||||
"generated_at": exported_at,
|
||||
"input_path": str(input_path.resolve()),
|
||||
"id_base": id_base,
|
||||
"total_cases": len(index_cases),
|
||||
"duplicate_names": duplicate_names,
|
||||
"cases": index_cases,
|
||||
}
|
||||
return payload, case_index
|
||||
|
||||
|
||||
def write_json(path: Path, payload: dict[str, Any]) -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(json.dumps(payload, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
|
||||
|
||||
|
||||
def main() -> int:
|
||||
args = parse_args()
|
||||
input_path = args.input.resolve()
|
||||
output_path = args.output.resolve()
|
||||
case_index_output = (
|
||||
args.case_index_output.resolve()
|
||||
if args.case_index_output is not None
|
||||
else output_path.with_name(f"{output_path.stem}.case_index.json")
|
||||
)
|
||||
|
||||
if args.id_base < 0:
|
||||
raise ValueError("--id-base must be greater than or equal to 0")
|
||||
if not input_path.is_file():
|
||||
raise FileNotFoundError(f"Input case list not found: {input_path}")
|
||||
|
||||
case_rows: list[dict[str, Any]] = []
|
||||
for line_no, raw_line in enumerate(input_path.read_text(encoding="utf-8").splitlines(), start=1):
|
||||
parsed = parse_case_line(raw_line, line_no)
|
||||
if parsed is None:
|
||||
continue
|
||||
case_name, source = parsed
|
||||
case_rows.append(
|
||||
{
|
||||
"line_no": line_no,
|
||||
"raw_line": raw_line,
|
||||
"case_name": case_name,
|
||||
"source": source,
|
||||
}
|
||||
)
|
||||
|
||||
if not case_rows:
|
||||
raise ValueError(f"No runnable cases were found in {input_path}")
|
||||
|
||||
payload, case_index = build_payload(
|
||||
input_path=input_path,
|
||||
case_rows=case_rows,
|
||||
id_base=args.id_base,
|
||||
)
|
||||
write_json(output_path, payload)
|
||||
write_json(case_index_output, case_index)
|
||||
|
||||
print(f"input: {input_path}")
|
||||
print(f"output: {output_path}")
|
||||
print(f"case_index_output: {case_index_output}")
|
||||
print(f"total_cases: {payload['summary']['total']}")
|
||||
print(f"duplicate_name_count: {payload['summary']['duplicate_name_count']}")
|
||||
print(f"pdcl_case_count: {payload['summary']['pdcl_case_count']}")
|
||||
print(f"standard_path_case_count: {payload['summary']['standard_path_case_count']}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
326
tools/feishu_project/case_calib_recovery.py
Executable file
326
tools/feishu_project/case_calib_recovery.py
Executable file
@@ -0,0 +1,326 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Recover camera4.json for downloaded standard-path issue cases."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any, Iterable, Optional
|
||||
|
||||
|
||||
TARGET_CALIB_REL = Path("test_data") / "calibs" / "camera4.json"
|
||||
CALIB_ATTACHMENT_NAMES = (
|
||||
"sigmastar.1/calibs/camera4.json",
|
||||
"test_data/calibs/camera4.json",
|
||||
"calibs/camera4.json",
|
||||
)
|
||||
CAMERA_CONFIG_FILE_NAME = "camera_config_folder.bin"
|
||||
PREFERRED_CAMERA_IDS = (4, 0)
|
||||
REQUIRED_FLAT_CALIB_KEYS = ("focal_u", "focal_v", "cu", "cv")
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ExtractedCalibSource:
|
||||
payload: dict[str, Any]
|
||||
source_kind: str
|
||||
source_path: Path
|
||||
detail: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CalibRecoveryResult:
|
||||
status: str
|
||||
detail: str
|
||||
target_path: Path
|
||||
source_path: Path | None = None
|
||||
source_kind: str | None = None
|
||||
|
||||
|
||||
def _safe_int(value: Any) -> int | None:
|
||||
try:
|
||||
if value is None:
|
||||
return None
|
||||
return int(value)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
def _sorted_unique_paths(paths: Iterable[Path]) -> list[Path]:
|
||||
unique = {path.resolve() if path.exists() else path for path in paths}
|
||||
return sorted(unique, key=lambda path: (len(path.parts), str(path)))
|
||||
|
||||
|
||||
def _camera_attachment_matches(name: str) -> bool:
|
||||
normalized = str(name).strip().replace("\\", "/")
|
||||
if normalized in CALIB_ATTACHMENT_NAMES:
|
||||
return True
|
||||
return normalized.endswith("/camera4.json") or normalized == "camera4.json"
|
||||
|
||||
|
||||
def iter_concatenated_json_objects(text: str) -> list[dict[str, Any]]:
|
||||
decoder = json.JSONDecoder()
|
||||
pos = 0
|
||||
payloads: list[dict[str, Any]] = []
|
||||
text_len = len(text)
|
||||
|
||||
while pos < text_len:
|
||||
while pos < text_len and (text[pos].isspace() or text[pos] == "\x00"):
|
||||
pos += 1
|
||||
if pos >= text_len:
|
||||
break
|
||||
|
||||
obj, end = decoder.raw_decode(text, pos)
|
||||
if isinstance(obj, dict):
|
||||
payloads.append(obj)
|
||||
pos = end
|
||||
|
||||
return payloads
|
||||
|
||||
|
||||
def _coerce_float_list(value: Any, *, min_len: int = 0) -> list[float]:
|
||||
if isinstance(value, dict):
|
||||
ordered = [value.get("x"), value.get("y"), value.get("z")]
|
||||
result = [float(item) for item in ordered if item is not None]
|
||||
elif isinstance(value, (list, tuple)):
|
||||
result = [float(item) for item in value]
|
||||
else:
|
||||
result = []
|
||||
while len(result) < min_len:
|
||||
result.append(0.0)
|
||||
return result
|
||||
|
||||
|
||||
def _is_valid_flat_calib_payload(payload: dict[str, Any]) -> bool:
|
||||
return all(payload.get(key) is not None for key in REQUIRED_FLAT_CALIB_KEYS)
|
||||
|
||||
|
||||
def _build_flat_camera4_payload(record: dict[str, Any], source_path: Path) -> dict[str, Any]:
|
||||
payload = dict(record)
|
||||
payload["focal_u"] = float(record["focal_u"])
|
||||
payload["focal_v"] = float(record["focal_v"])
|
||||
payload["cu"] = float(record["cu"])
|
||||
payload["cv"] = float(record["cv"])
|
||||
payload["roll"] = float(record.get("roll", 0.0))
|
||||
payload["pitch"] = float(record.get("pitch", 0.0))
|
||||
payload["yaw"] = float(record.get("yaw", 0.0))
|
||||
payload["distort_coeffs"] = _coerce_float_list(record.get("distort_coeffs"))
|
||||
payload["pos"] = _coerce_float_list(record.get("pos"), min_len=3)[:3]
|
||||
if payload.get("image_width") is not None:
|
||||
payload["image_width"] = int(payload["image_width"])
|
||||
if payload.get("image_height") is not None:
|
||||
payload["image_height"] = int(payload["image_height"])
|
||||
payload.setdefault("camera_id", 4)
|
||||
payload["recovered_from"] = CAMERA_CONFIG_FILE_NAME
|
||||
payload["recovered_from_path"] = str(source_path)
|
||||
return payload
|
||||
|
||||
|
||||
def extract_calib_from_camera_config_folder(
|
||||
config_path: Path,
|
||||
preferred_camera_ids: tuple[int, ...] = PREFERRED_CAMERA_IDS,
|
||||
) -> ExtractedCalibSource | None:
|
||||
text = config_path.read_text(encoding="utf-8")
|
||||
records = iter_concatenated_json_objects(text)
|
||||
valid_records = [record for record in records if _is_valid_flat_calib_payload(record)]
|
||||
if not valid_records:
|
||||
return None
|
||||
|
||||
selected_records = valid_records
|
||||
available_camera_ids = {
|
||||
_safe_int(record.get("camera_id"))
|
||||
for record in valid_records
|
||||
if _safe_int(record.get("camera_id")) is not None
|
||||
}
|
||||
chosen_camera_id: int | None = None
|
||||
|
||||
for camera_id in preferred_camera_ids:
|
||||
preferred_records = [
|
||||
record for record in valid_records if _safe_int(record.get("camera_id")) == camera_id
|
||||
]
|
||||
if preferred_records:
|
||||
selected_records = preferred_records
|
||||
chosen_camera_id = camera_id
|
||||
break
|
||||
|
||||
if chosen_camera_id is None:
|
||||
if len(available_camera_ids) == 1:
|
||||
chosen_camera_id = next(iter(available_camera_ids))
|
||||
else:
|
||||
counts: dict[int, int] = {}
|
||||
for record in valid_records:
|
||||
camera_id = _safe_int(record.get("camera_id"))
|
||||
if camera_id is None:
|
||||
continue
|
||||
counts[camera_id] = counts.get(camera_id, 0) + 1
|
||||
if counts:
|
||||
chosen_camera_id = max(counts.items(), key=lambda item: item[1])[0]
|
||||
selected_records = [
|
||||
record for record in valid_records if _safe_int(record.get("camera_id")) == chosen_camera_id
|
||||
]
|
||||
|
||||
best_record = max(
|
||||
enumerate(selected_records),
|
||||
key=lambda item: (_safe_int(item[1].get("utc_tick")) or -1, item[0]),
|
||||
)[1]
|
||||
payload = _build_flat_camera4_payload(best_record, config_path)
|
||||
detail = (
|
||||
f"camera_id={chosen_camera_id if chosen_camera_id is not None else 'unknown'} "
|
||||
f"records={len(valid_records)} latest_utc_tick={best_record.get('utc_tick')}"
|
||||
)
|
||||
return ExtractedCalibSource(
|
||||
payload=payload,
|
||||
source_kind="camera_config_folder",
|
||||
source_path=config_path,
|
||||
detail=detail,
|
||||
)
|
||||
|
||||
|
||||
def _extract_calib_from_mcap_with_clip_reader(mcap_path: Path) -> ExtractedCalibSource | None:
|
||||
try:
|
||||
from pdcl_pyclip.reader import ClipReader
|
||||
except ImportError:
|
||||
return None
|
||||
|
||||
reader = ClipReader(str(mcap_path))
|
||||
for attachment in reader.iter_attachments():
|
||||
if _camera_attachment_matches(attachment.name):
|
||||
payload = json.loads(attachment.data.decode("utf-8"))
|
||||
return ExtractedCalibSource(
|
||||
payload=payload,
|
||||
source_kind="mcap_attachment",
|
||||
source_path=mcap_path,
|
||||
detail=f"attachment={attachment.name}",
|
||||
)
|
||||
return None
|
||||
|
||||
|
||||
def _extract_calib_from_mcap_with_generic_reader(mcap_path: Path) -> ExtractedCalibSource | None:
|
||||
try:
|
||||
from mcap.reader import make_reader
|
||||
except ImportError:
|
||||
return None
|
||||
|
||||
with mcap_path.open("rb") as file:
|
||||
reader = make_reader(file)
|
||||
for attachment in reader.iter_attachments():
|
||||
if _camera_attachment_matches(attachment.name):
|
||||
payload = json.loads(attachment.data.decode("utf-8"))
|
||||
return ExtractedCalibSource(
|
||||
payload=payload,
|
||||
source_kind="mcap_attachment",
|
||||
source_path=mcap_path,
|
||||
detail=f"attachment={attachment.name}",
|
||||
)
|
||||
return None
|
||||
|
||||
|
||||
def extract_calib_from_mcap_attachment(mcap_path: Path) -> ExtractedCalibSource | None:
|
||||
errors: list[str] = []
|
||||
for extractor in (_extract_calib_from_mcap_with_clip_reader, _extract_calib_from_mcap_with_generic_reader):
|
||||
try:
|
||||
extracted = extractor(mcap_path)
|
||||
except Exception as exc: # pragma: no cover - recovery fallback logging
|
||||
errors.append(f"{extractor.__name__}: {type(exc).__name__}: {exc}")
|
||||
continue
|
||||
if extracted is not None:
|
||||
return extracted
|
||||
|
||||
if errors:
|
||||
raise RuntimeError("; ".join(errors))
|
||||
return None
|
||||
|
||||
|
||||
def _find_candidate_camera_config_paths(source_root: Path) -> list[Path]:
|
||||
return _sorted_unique_paths(source_root.rglob(CAMERA_CONFIG_FILE_NAME))
|
||||
|
||||
|
||||
def _find_candidate_mcap_paths(source_root: Path) -> list[Path]:
|
||||
return _sorted_unique_paths(source_root.rglob("*.mcap"))
|
||||
|
||||
|
||||
def _write_camera4_json(target_path: Path, payload: dict[str, Any]) -> None:
|
||||
target_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
target_path.write_text(
|
||||
json.dumps(payload, ensure_ascii=False, indent=2) + "\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
|
||||
def recover_camera4_json(
|
||||
source_root: Path,
|
||||
target_root: Path,
|
||||
dry_run: bool = False,
|
||||
) -> CalibRecoveryResult:
|
||||
target_path = target_root / TARGET_CALIB_REL
|
||||
if target_path.is_file():
|
||||
return CalibRecoveryResult(
|
||||
status="skipped_existing_calib",
|
||||
detail=f"target already exists: {target_path}",
|
||||
target_path=target_path,
|
||||
)
|
||||
|
||||
camera_config_paths = _find_candidate_camera_config_paths(source_root)
|
||||
mcap_paths = _find_candidate_mcap_paths(source_root)
|
||||
errors: list[str] = []
|
||||
|
||||
for config_path in camera_config_paths:
|
||||
try:
|
||||
extracted = extract_calib_from_camera_config_folder(config_path)
|
||||
except Exception as exc:
|
||||
errors.append(f"{config_path}: {type(exc).__name__}: {exc}")
|
||||
continue
|
||||
if extracted is None:
|
||||
continue
|
||||
if dry_run:
|
||||
return CalibRecoveryResult(
|
||||
status="planned_calib_recovery_from_camera_config_folder",
|
||||
detail=f"would write {target_path} from {config_path} ({extracted.detail})",
|
||||
target_path=target_path,
|
||||
source_path=config_path,
|
||||
source_kind=extracted.source_kind,
|
||||
)
|
||||
_write_camera4_json(target_path, extracted.payload)
|
||||
return CalibRecoveryResult(
|
||||
status="recovered_calib_from_camera_config_folder",
|
||||
detail=f"wrote {target_path} from {config_path} ({extracted.detail})",
|
||||
target_path=target_path,
|
||||
source_path=config_path,
|
||||
source_kind=extracted.source_kind,
|
||||
)
|
||||
|
||||
for mcap_path in mcap_paths:
|
||||
try:
|
||||
extracted = extract_calib_from_mcap_attachment(mcap_path)
|
||||
except Exception as exc:
|
||||
errors.append(f"{mcap_path}: {type(exc).__name__}: {exc}")
|
||||
continue
|
||||
if extracted is None:
|
||||
continue
|
||||
if dry_run:
|
||||
return CalibRecoveryResult(
|
||||
status="planned_calib_recovery_from_mcap_attachment",
|
||||
detail=f"would write {target_path} from {mcap_path} ({extracted.detail})",
|
||||
target_path=target_path,
|
||||
source_path=mcap_path,
|
||||
source_kind=extracted.source_kind,
|
||||
)
|
||||
_write_camera4_json(target_path, extracted.payload)
|
||||
return CalibRecoveryResult(
|
||||
status="recovered_calib_from_mcap_attachment",
|
||||
detail=f"wrote {target_path} from {mcap_path} ({extracted.detail})",
|
||||
target_path=target_path,
|
||||
source_path=mcap_path,
|
||||
source_kind=extracted.source_kind,
|
||||
)
|
||||
|
||||
checked_sources = (
|
||||
f"camera_config_folder.bin={len(camera_config_paths)} mcap={len(mcap_paths)}"
|
||||
)
|
||||
if errors:
|
||||
checked_sources += f"; errors: {' | '.join(errors[:3])}"
|
||||
return CalibRecoveryResult(
|
||||
status="failed_calib_recovery",
|
||||
detail=f"no recoverable calibration found under {source_root} ({checked_sources})",
|
||||
target_path=target_path,
|
||||
)
|
||||
9
tools/feishu_project/cncap_case.txt
Executable file
9
tools/feishu_project/cncap_case.txt
Executable file
@@ -0,0 +1,9 @@
|
||||
CPLA-夜晚_AEB_40_5 ADAS_S5STNF0T406280R_20260409152111002995_edda23ef80bd::20260408001902
|
||||
CSTA-LN_AEB_10_20 ADAS_S5STNF0T406280R_20260415153614565776_6304fc524d71::20260414173141
|
||||
CSTA-LN_AEB_10_20 ADAS_S5STNF0T406280R_20260415153614565776_7b6bcdb90977::20260414172819
|
||||
CSTA-LN_AEB_20_20 ADAS_S5STNF0T406280R_20260415153614565776_0ae4e25121ec::20260414171833
|
||||
CSTA-LN_AEB_20_20 ADAS_S5STNF0T406280R_20260415153614565776_c66607e24083::20260414170603
|
||||
CPLA-夜晚_AEB_FCW70_5 ADAS_S5STNF0T406280R_20260409152111002995_dd98b096d792::20260407233549
|
||||
CPLA-夜晚_AEB_FCW60_5 ADAS_S5STNF0T406280R_20260409152111002995_8065a4b59db6::20260407233159
|
||||
CBLA_AEB_FCW80_15 ADAS_S5STNF0T406280R_20260409152111002995_7254e457ef67::20260407115528
|
||||
CBLA_AEB_FCW70_15 ADAS_S5STNF0T406280R_20260409152111002995_44c27781ddc3::20260407114731
|
||||
838
tools/feishu_project/decode_issue_frame_window.py
Executable file
838
tools/feishu_project/decode_issue_frame_window.py
Executable file
@@ -0,0 +1,838 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Decode 61-frame windows around issue frame ids for D4Q2 network-share cases."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import io
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
from collections import Counter, deque
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Any, Iterable, Optional
|
||||
|
||||
import cv2
|
||||
import numpy as np
|
||||
|
||||
FILE = Path(__file__).resolve()
|
||||
ROOT = FILE.parents[2]
|
||||
if str(ROOT) not in sys.path:
|
||||
sys.path.append(str(ROOT))
|
||||
|
||||
from tools.model_inference.adapters.video_dir_inference_utils import (
|
||||
get_video_frame_info,
|
||||
read_video_frame_index,
|
||||
)
|
||||
|
||||
try:
|
||||
from pdcl_pyclip.decoder_struct import StructDecoder
|
||||
from pdcl_pyclip.msg_camera import VideoMessage
|
||||
from pdcl_pyclip.reader import ClipReader
|
||||
except ImportError:
|
||||
ClipReader = None
|
||||
StructDecoder = None
|
||||
VideoMessage = None
|
||||
|
||||
|
||||
NETWORK_SHARE_MARKERS = ("hfs.minieye.tech", "192.168.2.122")
|
||||
WINDOW_RADIUS = 30
|
||||
WINDOW_SIZE = WINDOW_RADIUS * 2 + 1
|
||||
DEFAULT_INPUT_JSON = ROOT / "tools" / "feishu_project" / "dongying_d4q2_zhibao_issue_list.json"
|
||||
DEFAULT_DOWNLOAD_ROOT = Path("/data1/dongying/Mono3d/D4Q2/feishu_project/downloaded_issue_data")
|
||||
DEFAULT_OUTPUT_ROOT = Path("/data1/dongying/Mono3d/D4Q2/feishu_project/decoded_issue_frame_windows")
|
||||
FRAME_ID_CAMERA4_RE = re.compile(r"camera4\s*:\s*(\d+)", re.IGNORECASE)
|
||||
FRAME_ID_ANY_CAMERA_RE = re.compile(r"(camera\d+)\s*:\s*(\d+)", re.IGNORECASE)
|
||||
PURE_DIGIT_RE = re.compile(r"^\d+$")
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class TargetFrame:
|
||||
camera: str
|
||||
frame_id: int
|
||||
raw_text: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CaseSource:
|
||||
issue_id: int
|
||||
issue_name: str
|
||||
issue_dir: Path
|
||||
path_dir: Path
|
||||
case_dir: Path
|
||||
relative_case_dir: Path
|
||||
target_frame: TargetFrame
|
||||
decode_all: bool
|
||||
source_mode: str
|
||||
source_paths: tuple[Path, ...]
|
||||
|
||||
|
||||
@dataclass
|
||||
class CaseResult:
|
||||
issue_id: int
|
||||
issue_name: str
|
||||
case_dir: str
|
||||
relative_case_dir: str
|
||||
target_camera: str
|
||||
target_frame_id: int | None
|
||||
decode_mode: str
|
||||
source_mode: str | None
|
||||
source_paths: list[str]
|
||||
output_dir: str
|
||||
status: str
|
||||
detail: str
|
||||
matched_field: str | None = None
|
||||
matched_frame_idx: int | None = None
|
||||
matched_topic: str | None = None
|
||||
extracted_count: int = 0
|
||||
|
||||
def to_dict(self) -> dict[str, Any]:
|
||||
return {
|
||||
"issue_id": self.issue_id,
|
||||
"issue_name": self.issue_name,
|
||||
"case_dir": self.case_dir,
|
||||
"relative_case_dir": self.relative_case_dir,
|
||||
"target_camera": self.target_camera,
|
||||
"target_frame_id": self.target_frame_id,
|
||||
"decode_mode": self.decode_mode,
|
||||
"source_mode": self.source_mode,
|
||||
"source_paths": self.source_paths,
|
||||
"output_dir": self.output_dir,
|
||||
"status": self.status,
|
||||
"detail": self.detail,
|
||||
"matched_field": self.matched_field,
|
||||
"matched_frame_idx": self.matched_frame_idx,
|
||||
"matched_topic": self.matched_topic,
|
||||
"extracted_count": self.extracted_count,
|
||||
}
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Decode 61-frame windows around issue frame ids for downloaded D4Q2 network-share cases."
|
||||
)
|
||||
parser.add_argument("--input-json", default=str(DEFAULT_INPUT_JSON))
|
||||
parser.add_argument("--download-root", default=str(DEFAULT_DOWNLOAD_ROOT))
|
||||
parser.add_argument("--output-root", default=str(DEFAULT_OUTPUT_ROOT))
|
||||
parser.add_argument("--manifest-path", default="")
|
||||
parser.add_argument("--issue-id", action="append", dest="issue_ids", type=int)
|
||||
parser.add_argument("--decode-all-issue-id", action="append", dest="decode_all_issue_ids", type=int)
|
||||
parser.add_argument("--window-radius", type=int, default=WINDOW_RADIUS)
|
||||
parser.add_argument("--jpg-quality", type=int, default=95)
|
||||
parser.add_argument("--dry-run", action="store_true")
|
||||
parser.add_argument("--skip-existing", action="store_true")
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def ensure_dir(path: Path, dry_run: bool) -> None:
|
||||
if dry_run:
|
||||
return
|
||||
path.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
|
||||
def load_issue_items(path: Path) -> list[dict[str, Any]]:
|
||||
payload = json.loads(path.read_text(encoding="utf-8"))
|
||||
return payload["items"]
|
||||
|
||||
|
||||
def parse_target_frame(frame_text: object) -> TargetFrame | None:
|
||||
text = "" if frame_text is None else str(frame_text).strip()
|
||||
if not text:
|
||||
return None
|
||||
camera4_match = FRAME_ID_CAMERA4_RE.search(text)
|
||||
if camera4_match:
|
||||
return TargetFrame(camera="camera4", frame_id=int(camera4_match.group(1)), raw_text=text)
|
||||
if PURE_DIGIT_RE.fullmatch(text):
|
||||
return TargetFrame(camera="any", frame_id=int(text), raw_text=text)
|
||||
any_camera_match = FRAME_ID_ANY_CAMERA_RE.search(text)
|
||||
if any_camera_match:
|
||||
return TargetFrame(camera=any_camera_match.group(1).lower(), frame_id=int(any_camera_match.group(2)), raw_text=text)
|
||||
return None
|
||||
|
||||
|
||||
def is_network_share_issue(item: dict[str, Any]) -> bool:
|
||||
address = str(item.get("问题数据地址") or "")
|
||||
return any(marker in address for marker in NETWORK_SHARE_MARKERS)
|
||||
|
||||
|
||||
def find_candidate_mcaps(case_dir: Path) -> list[Path]:
|
||||
root_level = sorted(
|
||||
path
|
||||
for path in case_dir.iterdir()
|
||||
if path.is_file() and path.suffix.lower() == ".mcap" and "_PB_" not in path.name
|
||||
)
|
||||
if root_level:
|
||||
return root_level
|
||||
|
||||
recursive = sorted(
|
||||
path
|
||||
for path in case_dir.rglob("*.mcap")
|
||||
if path.is_file() and "_PB_" not in path.name
|
||||
)
|
||||
return recursive
|
||||
|
||||
|
||||
def find_camera4_bin(case_dir: Path) -> Path | None:
|
||||
candidates = sorted(case_dir.rglob("camera4.bin"), key=lambda p: (len(p.relative_to(case_dir).parts), str(p)))
|
||||
return candidates[0] if candidates else None
|
||||
|
||||
|
||||
def discover_case_sources(
|
||||
items: list[dict[str, Any]],
|
||||
download_root: Path,
|
||||
issue_filter: set[int] | None,
|
||||
decode_all_issue_filter: set[int],
|
||||
) -> tuple[list[CaseSource], list[CaseResult]]:
|
||||
discovered: list[CaseSource] = []
|
||||
skipped: list[CaseResult] = []
|
||||
|
||||
for item in items:
|
||||
issue_id = int(item["id"])
|
||||
if issue_filter and issue_id not in issue_filter:
|
||||
continue
|
||||
if not is_network_share_issue(item):
|
||||
continue
|
||||
|
||||
issue_name = str(item["name"])
|
||||
issue_dir = download_root / f"issue_{issue_id}"
|
||||
target_frame = parse_target_frame(item.get("问题发生frameid"))
|
||||
if target_frame is None:
|
||||
skipped.append(
|
||||
CaseResult(
|
||||
issue_id=issue_id,
|
||||
issue_name=issue_name,
|
||||
case_dir="",
|
||||
relative_case_dir="",
|
||||
target_camera="",
|
||||
target_frame_id=None,
|
||||
decode_mode="window",
|
||||
source_mode=None,
|
||||
source_paths=[],
|
||||
output_dir="",
|
||||
status="skipped_missing_frame_id",
|
||||
detail=f"unparseable frame id: {item.get('问题发生frameid')!r}",
|
||||
)
|
||||
)
|
||||
continue
|
||||
|
||||
if target_frame.camera not in {"camera4", "camera1", "any"}:
|
||||
skipped.append(
|
||||
CaseResult(
|
||||
issue_id=issue_id,
|
||||
issue_name=issue_name,
|
||||
case_dir="",
|
||||
relative_case_dir="",
|
||||
target_camera=target_frame.camera,
|
||||
target_frame_id=target_frame.frame_id,
|
||||
decode_mode="all" if issue_id in decode_all_issue_filter else "window",
|
||||
source_mode=None,
|
||||
source_paths=[],
|
||||
output_dir="",
|
||||
status="skipped_unsupported_camera",
|
||||
detail=f"unsupported camera selector in frame id: {target_frame.raw_text}",
|
||||
)
|
||||
)
|
||||
continue
|
||||
|
||||
if not issue_dir.is_dir():
|
||||
skipped.append(
|
||||
CaseResult(
|
||||
issue_id=issue_id,
|
||||
issue_name=issue_name,
|
||||
case_dir=str(issue_dir),
|
||||
relative_case_dir=str(issue_dir.name),
|
||||
target_camera=target_frame.camera,
|
||||
target_frame_id=target_frame.frame_id,
|
||||
decode_mode="all" if issue_id in decode_all_issue_filter else "window",
|
||||
source_mode=None,
|
||||
source_paths=[],
|
||||
output_dir="",
|
||||
status="skipped_missing_issue_dir",
|
||||
detail=f"issue download directory not found: {issue_dir}",
|
||||
)
|
||||
)
|
||||
continue
|
||||
|
||||
path_dirs = sorted(path for path in issue_dir.iterdir() if path.is_dir() and path.name.startswith("path_"))
|
||||
if not path_dirs:
|
||||
skipped.append(
|
||||
CaseResult(
|
||||
issue_id=issue_id,
|
||||
issue_name=issue_name,
|
||||
case_dir=str(issue_dir),
|
||||
relative_case_dir=str(issue_dir.name),
|
||||
target_camera=target_frame.camera,
|
||||
target_frame_id=target_frame.frame_id,
|
||||
decode_mode="all" if issue_id in decode_all_issue_filter else "window",
|
||||
source_mode=None,
|
||||
source_paths=[],
|
||||
output_dir="",
|
||||
status="skipped_no_path_cases",
|
||||
detail="no path_* directories found for network-share issue",
|
||||
)
|
||||
)
|
||||
continue
|
||||
|
||||
for path_dir in path_dirs:
|
||||
case_dirs = sorted(path for path in path_dir.iterdir() if path.is_dir())
|
||||
if not case_dirs:
|
||||
skipped.append(
|
||||
CaseResult(
|
||||
issue_id=issue_id,
|
||||
issue_name=issue_name,
|
||||
case_dir=str(path_dir),
|
||||
relative_case_dir=str(path_dir.relative_to(download_root)),
|
||||
target_camera=target_frame.camera,
|
||||
target_frame_id=target_frame.frame_id,
|
||||
decode_mode="all" if issue_id in decode_all_issue_filter else "window",
|
||||
source_mode=None,
|
||||
source_paths=[],
|
||||
output_dir="",
|
||||
status="skipped_empty_path_dir",
|
||||
detail="path_* directory does not contain any case subdirectory",
|
||||
)
|
||||
)
|
||||
continue
|
||||
|
||||
for case_dir in case_dirs:
|
||||
relative_case_dir = case_dir.relative_to(download_root)
|
||||
candidate_mcaps = find_candidate_mcaps(case_dir)
|
||||
if candidate_mcaps:
|
||||
discovered.append(
|
||||
CaseSource(
|
||||
issue_id=issue_id,
|
||||
issue_name=issue_name,
|
||||
issue_dir=issue_dir,
|
||||
path_dir=path_dir,
|
||||
case_dir=case_dir,
|
||||
relative_case_dir=relative_case_dir,
|
||||
target_frame=target_frame,
|
||||
decode_all=issue_id in decode_all_issue_filter,
|
||||
source_mode="mcap_stream",
|
||||
source_paths=tuple(candidate_mcaps),
|
||||
)
|
||||
)
|
||||
continue
|
||||
|
||||
camera4_bin = find_camera4_bin(case_dir)
|
||||
if camera4_bin is not None:
|
||||
discovered.append(
|
||||
CaseSource(
|
||||
issue_id=issue_id,
|
||||
issue_name=issue_name,
|
||||
issue_dir=issue_dir,
|
||||
path_dir=path_dir,
|
||||
case_dir=case_dir,
|
||||
relative_case_dir=relative_case_dir,
|
||||
target_frame=target_frame,
|
||||
decode_all=issue_id in decode_all_issue_filter,
|
||||
source_mode="camera4_bin",
|
||||
source_paths=(camera4_bin,),
|
||||
)
|
||||
)
|
||||
continue
|
||||
|
||||
skipped.append(
|
||||
CaseResult(
|
||||
issue_id=issue_id,
|
||||
issue_name=issue_name,
|
||||
case_dir=str(case_dir),
|
||||
relative_case_dir=str(relative_case_dir),
|
||||
target_camera=target_frame.camera,
|
||||
target_frame_id=target_frame.frame_id,
|
||||
decode_mode="all" if issue_id in decode_all_issue_filter else "window",
|
||||
source_mode=None,
|
||||
source_paths=[],
|
||||
output_dir="",
|
||||
status="skipped_no_source",
|
||||
detail="no non-PB .mcap or camera4.bin found under case directory",
|
||||
)
|
||||
)
|
||||
return discovered, skipped
|
||||
|
||||
|
||||
def _plane_to_ndarray(plane) -> np.ndarray:
|
||||
stride = plane.line_size
|
||||
height = plane.height
|
||||
width = plane.width
|
||||
array = np.frombuffer(plane, dtype=np.uint8)
|
||||
if stride == width:
|
||||
return array.reshape(height, width)
|
||||
return array.reshape(height, stride)[:, :width]
|
||||
|
||||
|
||||
def _h265_payload_to_bgr(payload: bytes) -> np.ndarray:
|
||||
try:
|
||||
import av
|
||||
except ImportError as exc:
|
||||
raise ImportError("PyAV is required for MCAP frame decoding.") from exc
|
||||
|
||||
container = av.open(io.BytesIO(payload))
|
||||
for frame in container.decode(video=0):
|
||||
y_plane = _plane_to_ndarray(frame.planes[0])
|
||||
u_plane = _plane_to_ndarray(frame.planes[1])
|
||||
v_plane = _plane_to_ndarray(frame.planes[2])
|
||||
uv_plane = np.zeros((u_plane.shape[0], u_plane.shape[1] * 2), dtype=np.uint8)
|
||||
uv_plane[:, 0::2] = u_plane
|
||||
uv_plane[:, 1::2] = v_plane
|
||||
yuv_image = np.concatenate((y_plane.copy(), uv_plane), axis=0)
|
||||
return cv2.cvtColor(yuv_image, cv2.COLOR_YUV2BGR_NV12)
|
||||
raise ValueError("decode failed: no video frame in payload")
|
||||
|
||||
|
||||
def iter_mcap_frames(mcap_paths: Iterable[Path], topic_candidates: tuple[str, ...]) -> Iterable[dict[str, Any]]:
|
||||
if ClipReader is None or StructDecoder is None or VideoMessage is None:
|
||||
raise ImportError("pdcl_pyclip is required for MCAP extraction.")
|
||||
|
||||
decoder = StructDecoder()
|
||||
for mcap_path in mcap_paths:
|
||||
reader = ClipReader(str(mcap_path))
|
||||
for schema, channel, msg in reader.iter_messages():
|
||||
data = decoder.decode(schema, channel, msg)
|
||||
if not isinstance(data, VideoMessage):
|
||||
continue
|
||||
frame_id = getattr(data, "frame_id", None)
|
||||
if frame_id is None:
|
||||
continue
|
||||
yield {
|
||||
"source_path": mcap_path,
|
||||
"topic": getattr(channel, "topic", ""),
|
||||
"frame_id": int(frame_id),
|
||||
"payload": data.payload,
|
||||
"timestamp": str(getattr(data, "timestamp", "")),
|
||||
}
|
||||
|
||||
|
||||
def topic_matches_camera(topic: str, target_camera: str) -> bool:
|
||||
normalized_topic = str(topic or "").lower()
|
||||
if target_camera == "any":
|
||||
return "camera" in normalized_topic
|
||||
return target_camera in normalized_topic
|
||||
|
||||
|
||||
def collect_mcap_window(
|
||||
mcap_paths: tuple[Path, ...],
|
||||
target_camera: str,
|
||||
target_frame_id: int,
|
||||
window_radius: int,
|
||||
) -> tuple[list[dict[str, Any]], str, str]:
|
||||
buffer_before: deque[dict[str, Any]] = deque(maxlen=window_radius)
|
||||
selected: list[dict[str, Any]] = []
|
||||
trailing_needed = window_radius
|
||||
found = False
|
||||
matched_topic = ""
|
||||
|
||||
for frame in iter_mcap_frames(mcap_paths, tuple()):
|
||||
if not topic_matches_camera(frame["topic"], target_camera):
|
||||
continue
|
||||
if not found:
|
||||
if frame["frame_id"] == target_frame_id:
|
||||
selected = list(buffer_before) + [frame]
|
||||
found = True
|
||||
matched_topic = str(frame["topic"])
|
||||
else:
|
||||
buffer_before.append(frame)
|
||||
else:
|
||||
selected.append(frame)
|
||||
trailing_needed -= 1
|
||||
if trailing_needed <= 0:
|
||||
break
|
||||
|
||||
if not found:
|
||||
raise FileNotFoundError(f"target frame_id {target_frame_id} not found in MCAP sources for camera selector {target_camera}")
|
||||
|
||||
return selected, "frame_id", matched_topic
|
||||
|
||||
|
||||
def collect_mcap_all_frames(
|
||||
mcap_paths: tuple[Path, ...],
|
||||
target_camera: str,
|
||||
) -> tuple[list[dict[str, Any]], str]:
|
||||
selected: list[dict[str, Any]] = []
|
||||
matched_topic = ""
|
||||
for frame in iter_mcap_frames(mcap_paths, tuple()):
|
||||
if not topic_matches_camera(frame["topic"], target_camera):
|
||||
continue
|
||||
if not matched_topic:
|
||||
matched_topic = str(frame["topic"])
|
||||
selected.append(frame)
|
||||
|
||||
if not selected:
|
||||
raise FileNotFoundError(f"no MCAP frames found for camera selector {target_camera}")
|
||||
|
||||
return selected, matched_topic
|
||||
|
||||
|
||||
def find_video_frame_match(index_payload: Optional[dict[str, Any]], target_frame_id: int) -> tuple[int | None, str | None]:
|
||||
if not index_payload:
|
||||
return None, None
|
||||
|
||||
fields = index_payload.get("fields", {}) or {}
|
||||
index_list = index_payload.get("index", []) or []
|
||||
for field_name in ("frame_id", "cve_frame_id"):
|
||||
if field_name not in fields:
|
||||
continue
|
||||
for frame_idx in range(len(index_list)):
|
||||
info = get_video_frame_info(index_payload, frame_idx)
|
||||
if info is None:
|
||||
continue
|
||||
value = info.get(field_name)
|
||||
try:
|
||||
if int(value) == target_frame_id:
|
||||
return frame_idx, field_name
|
||||
except (TypeError, ValueError):
|
||||
continue
|
||||
return None, None
|
||||
|
||||
|
||||
def collect_video_window(
|
||||
video_path: Path,
|
||||
target_frame_id: int,
|
||||
window_radius: int,
|
||||
) -> tuple[list[dict[str, Any]], str, int]:
|
||||
index_payload = read_video_frame_index(video_path)
|
||||
matched_frame_idx, matched_field = find_video_frame_match(index_payload, target_frame_id)
|
||||
if matched_frame_idx is None or matched_field is None:
|
||||
raise FileNotFoundError(f"target frame_id {target_frame_id} not found in video index")
|
||||
|
||||
start_idx = max(0, matched_frame_idx - window_radius)
|
||||
end_idx = matched_frame_idx + window_radius
|
||||
|
||||
cap = cv2.VideoCapture(str(video_path))
|
||||
if not cap.isOpened():
|
||||
raise RuntimeError(f"failed to open video file: {video_path}")
|
||||
|
||||
selected: list[dict[str, Any]] = []
|
||||
try:
|
||||
cap.set(cv2.CAP_PROP_POS_FRAMES, start_idx)
|
||||
current_idx = start_idx
|
||||
while current_idx <= end_idx:
|
||||
ret, frame = cap.read()
|
||||
if not ret:
|
||||
break
|
||||
frame_info = get_video_frame_info(index_payload, current_idx) if index_payload else None
|
||||
selected.append(
|
||||
{
|
||||
"source_path": video_path,
|
||||
"frame_idx": current_idx,
|
||||
"frame_id": None if frame_info is None else frame_info.get("frame_id"),
|
||||
"cve_frame_id": None if frame_info is None else frame_info.get("cve_frame_id"),
|
||||
"timestamp": "" if frame_info is None else str(frame_info.get("timestamp", "")),
|
||||
"image": frame,
|
||||
}
|
||||
)
|
||||
current_idx += 1
|
||||
finally:
|
||||
cap.release()
|
||||
|
||||
return selected, matched_field, matched_frame_idx
|
||||
|
||||
|
||||
def collect_video_all_frames(video_path: Path) -> tuple[list[dict[str, Any]], int]:
|
||||
index_payload = read_video_frame_index(video_path)
|
||||
cap = cv2.VideoCapture(str(video_path))
|
||||
if not cap.isOpened():
|
||||
raise RuntimeError(f"failed to open video file: {video_path}")
|
||||
|
||||
selected: list[dict[str, Any]] = []
|
||||
current_idx = 0
|
||||
try:
|
||||
while True:
|
||||
ret, frame = cap.read()
|
||||
if not ret:
|
||||
break
|
||||
frame_info = get_video_frame_info(index_payload, current_idx) if index_payload else None
|
||||
selected.append(
|
||||
{
|
||||
"source_path": video_path,
|
||||
"frame_idx": current_idx,
|
||||
"frame_id": None if frame_info is None else frame_info.get("frame_id"),
|
||||
"cve_frame_id": None if frame_info is None else frame_info.get("cve_frame_id"),
|
||||
"timestamp": "" if frame_info is None else str(frame_info.get("timestamp", "")),
|
||||
"image": frame,
|
||||
}
|
||||
)
|
||||
current_idx += 1
|
||||
finally:
|
||||
cap.release()
|
||||
|
||||
if not selected:
|
||||
raise FileNotFoundError(f"no readable frames found in video file: {video_path}")
|
||||
return selected, len(selected)
|
||||
|
||||
|
||||
def save_decoded_frames(
|
||||
issue_id: int,
|
||||
output_dir: Path,
|
||||
frames: list[dict[str, Any]],
|
||||
target_frame_id: int,
|
||||
window_radius: int,
|
||||
jpg_quality: int,
|
||||
dry_run: bool,
|
||||
decode_all: bool,
|
||||
) -> int:
|
||||
images_dir = output_dir / ("frames_all" if decode_all else "frames_window")
|
||||
if dry_run:
|
||||
return len(frames)
|
||||
|
||||
ensure_dir(images_dir, dry_run=False)
|
||||
encode_params = [int(cv2.IMWRITE_JPEG_QUALITY), int(jpg_quality)]
|
||||
target_index = None if decode_all else min(window_radius, len(frames) - 1)
|
||||
for index, frame in enumerate(frames):
|
||||
offset = 0 if target_index is None else index - target_index
|
||||
frame_id = frame.get("frame_id")
|
||||
if frame_id is None:
|
||||
frame_id = frame.get("cve_frame_id")
|
||||
frame_id_token = "na" if frame_id is None else str(frame_id)
|
||||
topic = str(frame.get("topic") or "")
|
||||
camera_id_match = re.search(r"camera(\d+)", topic, re.IGNORECASE)
|
||||
if camera_id_match:
|
||||
camera_id_token = f"camera{camera_id_match.group(1)}"
|
||||
else:
|
||||
source_path = Path(str(frame.get("source_path") or ""))
|
||||
camera_id_token = source_path.stem if source_path.stem else "camera"
|
||||
if decode_all:
|
||||
filename = f"{issue_id}_{camera_id_token}_{frame_id_token}.jpg"
|
||||
else:
|
||||
filename = f"{issue_id}_{camera_id_token}_{frame_id_token}.jpg"
|
||||
|
||||
image = frame.get("image")
|
||||
if image is None:
|
||||
image = _h265_payload_to_bgr(frame["payload"])
|
||||
if not cv2.imwrite(str(images_dir / filename), image, encode_params):
|
||||
raise IOError(f"failed to write image: {images_dir / filename}")
|
||||
return len(frames)
|
||||
|
||||
|
||||
def process_case(case: CaseSource, output_root: Path, window_radius: int, jpg_quality: int, dry_run: bool, skip_existing: bool) -> CaseResult:
|
||||
output_dir = output_root / case.relative_case_dir
|
||||
images_dir = output_dir / ("frames_all" if case.decode_all else "frames_window")
|
||||
if skip_existing and images_dir.is_dir():
|
||||
existing_images = [path for path in images_dir.iterdir() if path.is_file()]
|
||||
if existing_images:
|
||||
return CaseResult(
|
||||
issue_id=case.issue_id,
|
||||
issue_name=case.issue_name,
|
||||
case_dir=str(case.case_dir),
|
||||
relative_case_dir=str(case.relative_case_dir),
|
||||
target_camera=case.target_frame.camera,
|
||||
target_frame_id=case.target_frame.frame_id,
|
||||
decode_mode="all" if case.decode_all else "window",
|
||||
source_mode=case.source_mode,
|
||||
source_paths=[str(path) for path in case.source_paths],
|
||||
output_dir=str(output_dir),
|
||||
status="skipped_existing",
|
||||
detail="existing extracted frames found",
|
||||
extracted_count=len(existing_images),
|
||||
)
|
||||
|
||||
try:
|
||||
if case.source_mode == "mcap_stream":
|
||||
try:
|
||||
if case.decode_all:
|
||||
frames, matched_topic = collect_mcap_all_frames(case.source_paths, case.target_frame.camera)
|
||||
matched_field = None
|
||||
else:
|
||||
frames, matched_field, matched_topic = collect_mcap_window(
|
||||
case.source_paths,
|
||||
case.target_frame.camera,
|
||||
case.target_frame.frame_id,
|
||||
window_radius,
|
||||
)
|
||||
extracted_count = save_decoded_frames(
|
||||
case.issue_id, output_dir, frames, case.target_frame.frame_id, window_radius, jpg_quality, dry_run, case.decode_all
|
||||
)
|
||||
detail = (
|
||||
f"decoded {extracted_count} frames from {len(case.source_paths)} mcap file(s)"
|
||||
if case.decode_all
|
||||
else f"decoded {extracted_count} frames from {len(case.source_paths)} mcap file(s)"
|
||||
)
|
||||
return CaseResult(
|
||||
issue_id=case.issue_id,
|
||||
issue_name=case.issue_name,
|
||||
case_dir=str(case.case_dir),
|
||||
relative_case_dir=str(case.relative_case_dir),
|
||||
target_camera=case.target_frame.camera,
|
||||
target_frame_id=case.target_frame.frame_id,
|
||||
decode_mode="all" if case.decode_all else "window",
|
||||
source_mode=case.source_mode,
|
||||
source_paths=[str(path) for path in case.source_paths],
|
||||
output_dir=str(output_dir),
|
||||
status="planned" if dry_run else "decoded_mcap",
|
||||
detail=("would " + detail) if dry_run else detail,
|
||||
matched_field=matched_field,
|
||||
matched_frame_idx=None,
|
||||
matched_topic=matched_topic,
|
||||
extracted_count=extracted_count,
|
||||
)
|
||||
except Exception as mcap_exc:
|
||||
fallback_camera4 = find_camera4_bin(case.case_dir)
|
||||
if fallback_camera4 is None or case.target_frame.camera not in {"camera4", "any"}:
|
||||
raise mcap_exc
|
||||
|
||||
if case.decode_all:
|
||||
frames, extracted_count_all = collect_video_all_frames(fallback_camera4)
|
||||
matched_field = None
|
||||
matched_frame_idx = None
|
||||
extracted_count = save_decoded_frames(
|
||||
case.issue_id, output_dir, frames, case.target_frame.frame_id, window_radius, jpg_quality, dry_run, case.decode_all
|
||||
)
|
||||
else:
|
||||
frames, matched_field, matched_frame_idx = collect_video_window(
|
||||
fallback_camera4,
|
||||
case.target_frame.frame_id,
|
||||
window_radius,
|
||||
)
|
||||
extracted_count = save_decoded_frames(
|
||||
case.issue_id, output_dir, frames, case.target_frame.frame_id, window_radius, jpg_quality, dry_run, case.decode_all
|
||||
)
|
||||
detail = (
|
||||
f"decoded {extracted_count} frames from camera4.bin fallback after mcap lookup failed: "
|
||||
f"{type(mcap_exc).__name__}: {mcap_exc}"
|
||||
)
|
||||
return CaseResult(
|
||||
issue_id=case.issue_id,
|
||||
issue_name=case.issue_name,
|
||||
case_dir=str(case.case_dir),
|
||||
relative_case_dir=str(case.relative_case_dir),
|
||||
target_camera=case.target_frame.camera,
|
||||
target_frame_id=case.target_frame.frame_id,
|
||||
decode_mode="all" if case.decode_all else "window",
|
||||
source_mode="camera4_bin_fallback",
|
||||
source_paths=[str(path) for path in case.source_paths] + [str(fallback_camera4)],
|
||||
output_dir=str(output_dir),
|
||||
status="planned" if dry_run else "decoded_camera4_bin_fallback",
|
||||
detail=("would " + detail) if dry_run else detail,
|
||||
matched_field=matched_field,
|
||||
matched_frame_idx=matched_frame_idx,
|
||||
matched_topic=None,
|
||||
extracted_count=extracted_count,
|
||||
)
|
||||
|
||||
if case.source_mode == "camera4_bin":
|
||||
if case.decode_all:
|
||||
frames, _ = collect_video_all_frames(case.source_paths[0])
|
||||
matched_field, matched_frame_idx = None, None
|
||||
else:
|
||||
frames, matched_field, matched_frame_idx = collect_video_window(case.source_paths[0], case.target_frame.frame_id, window_radius)
|
||||
extracted_count = save_decoded_frames(
|
||||
case.issue_id, output_dir, frames, case.target_frame.frame_id, window_radius, jpg_quality, dry_run, case.decode_all
|
||||
)
|
||||
detail = f"decoded {extracted_count} frames from camera4.bin"
|
||||
return CaseResult(
|
||||
issue_id=case.issue_id,
|
||||
issue_name=case.issue_name,
|
||||
case_dir=str(case.case_dir),
|
||||
relative_case_dir=str(case.relative_case_dir),
|
||||
target_camera=case.target_frame.camera,
|
||||
target_frame_id=case.target_frame.frame_id,
|
||||
decode_mode="all" if case.decode_all else "window",
|
||||
source_mode=case.source_mode,
|
||||
source_paths=[str(path) for path in case.source_paths],
|
||||
output_dir=str(output_dir),
|
||||
status="planned" if dry_run else "decoded_camera4_bin",
|
||||
detail=("would " + detail) if dry_run else detail,
|
||||
matched_field=matched_field,
|
||||
matched_frame_idx=matched_frame_idx,
|
||||
matched_topic=None,
|
||||
extracted_count=extracted_count,
|
||||
)
|
||||
|
||||
return CaseResult(
|
||||
issue_id=case.issue_id,
|
||||
issue_name=case.issue_name,
|
||||
case_dir=str(case.case_dir),
|
||||
relative_case_dir=str(case.relative_case_dir),
|
||||
target_camera=case.target_frame.camera,
|
||||
target_frame_id=case.target_frame.frame_id,
|
||||
decode_mode="all" if case.decode_all else "window",
|
||||
source_mode=case.source_mode,
|
||||
source_paths=[str(path) for path in case.source_paths],
|
||||
output_dir=str(output_dir),
|
||||
status="skipped_no_source",
|
||||
detail=f"unsupported source mode: {case.source_mode}",
|
||||
)
|
||||
except Exception as exc:
|
||||
return CaseResult(
|
||||
issue_id=case.issue_id,
|
||||
issue_name=case.issue_name,
|
||||
case_dir=str(case.case_dir),
|
||||
relative_case_dir=str(case.relative_case_dir),
|
||||
target_camera=case.target_frame.camera,
|
||||
target_frame_id=case.target_frame.frame_id,
|
||||
decode_mode="all" if case.decode_all else "window",
|
||||
source_mode=case.source_mode,
|
||||
source_paths=[str(path) for path in case.source_paths],
|
||||
output_dir=str(output_dir),
|
||||
status="failed",
|
||||
detail=f"{type(exc).__name__}: {exc}",
|
||||
)
|
||||
|
||||
|
||||
def build_manifest(args: argparse.Namespace, input_json: Path, download_root: Path, output_root: Path, discovered: list[CaseSource], results: list[CaseResult]) -> dict[str, Any]:
|
||||
summary = Counter(result.status for result in results)
|
||||
return {
|
||||
"generated_at": datetime.now().astimezone().isoformat(timespec="seconds"),
|
||||
"input_json": str(input_json),
|
||||
"download_root": str(download_root),
|
||||
"output_root": str(output_root),
|
||||
"issue_filter": args.issue_ids or [],
|
||||
"decode_all_issue_filter": args.decode_all_issue_ids or [],
|
||||
"window_radius": args.window_radius,
|
||||
"window_size": args.window_radius * 2 + 1,
|
||||
"jpg_quality": args.jpg_quality,
|
||||
"dry_run": args.dry_run,
|
||||
"skip_existing": args.skip_existing,
|
||||
"total_discovered_cases": len(discovered),
|
||||
"summary": dict(summary),
|
||||
"cases": [result.to_dict() for result in results],
|
||||
}
|
||||
|
||||
|
||||
def print_summary(manifest: dict[str, Any]) -> None:
|
||||
print(f"input_json: {manifest['input_json']}")
|
||||
print(f"download_root: {manifest['download_root']}")
|
||||
print(f"output_root: {manifest['output_root']}")
|
||||
print(f"window_size: {manifest['window_size']}")
|
||||
print(f"dry_run: {manifest['dry_run']}")
|
||||
print(f"total_discovered_cases: {manifest['total_discovered_cases']}")
|
||||
for status, count in sorted(manifest["summary"].items()):
|
||||
print(f"{status}: {count}")
|
||||
|
||||
|
||||
def main() -> int:
|
||||
args = parse_args()
|
||||
input_json = Path(args.input_json).resolve()
|
||||
download_root = Path(args.download_root).resolve()
|
||||
output_root = Path(args.output_root).resolve()
|
||||
manifest_path = (
|
||||
Path(args.manifest_path).resolve()
|
||||
if args.manifest_path
|
||||
else output_root / "decode_manifest.json"
|
||||
)
|
||||
|
||||
items = load_issue_items(input_json)
|
||||
issue_filter = set(args.issue_ids) if args.issue_ids else None
|
||||
decode_all_issue_filter = set(args.decode_all_issue_ids or [])
|
||||
discovered, skipped = discover_case_sources(items, download_root, issue_filter, decode_all_issue_filter)
|
||||
results = list(skipped)
|
||||
for case in discovered:
|
||||
results.append(process_case(case, output_root, args.window_radius, args.jpg_quality, args.dry_run, args.skip_existing))
|
||||
|
||||
manifest = build_manifest(args, input_json, download_root, output_root, discovered, results)
|
||||
if not args.dry_run:
|
||||
ensure_dir(manifest_path.parent, dry_run=False)
|
||||
manifest_path.write_text(json.dumps(manifest, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
|
||||
|
||||
print_summary(manifest)
|
||||
if args.dry_run:
|
||||
print(f"manifest (not written in dry-run): {manifest_path}")
|
||||
else:
|
||||
print(f"manifest: {manifest_path}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
45
tools/feishu_project/decode_issue_frame_window.sh
Executable file
45
tools/feishu_project/decode_issue_frame_window.sh
Executable file
@@ -0,0 +1,45 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
|
||||
PROJECT_ROOT=$(cd "${SCRIPT_DIR}/../.." && pwd)
|
||||
|
||||
INPUT_JSON=${INPUT_JSON:-"${PROJECT_ROOT}/tools/feishu_project/dongying_d4q2_zhibao_issue_list.json"}
|
||||
DOWNLOAD_ROOT=${DOWNLOAD_ROOT:-/data1/dongying/Mono3d/D4Q2/feishu_project/downloaded_issue_data}
|
||||
OUTPUT_ROOT=${OUTPUT_ROOT:-/data1/dongying/Mono3d/D4Q2/feishu_project/decoded_issue_frame_windows_v2}
|
||||
MANIFEST_PATH=${MANIFEST_PATH:-"${OUTPUT_ROOT}/decode_manifest.json"}
|
||||
PYTHON_BIN=${PYTHON_BIN:-/deeplearning_team/ydong/dongying/miniconda/envs/dev/bin/python}
|
||||
WINDOW_RADIUS=${WINDOW_RADIUS:-30}
|
||||
JPG_QUALITY=${JPG_QUALITY:-95}
|
||||
DRY_RUN=${DRY_RUN:-0}
|
||||
SKIP_EXISTING=${SKIP_EXISTING:-0}
|
||||
DECODE_ALL_ISSUE_IDS=${DECODE_ALL_ISSUE_IDS:-}
|
||||
|
||||
CMD=(
|
||||
"${PYTHON_BIN}" "${PROJECT_ROOT}/tools/feishu_project/decode_issue_frame_window.py"
|
||||
--input-json "${INPUT_JSON}"
|
||||
--download-root "${DOWNLOAD_ROOT}"
|
||||
--output-root "${OUTPUT_ROOT}"
|
||||
--manifest-path "${MANIFEST_PATH}"
|
||||
--window-radius "${WINDOW_RADIUS}"
|
||||
--jpg-quality "${JPG_QUALITY}"
|
||||
)
|
||||
|
||||
if [[ "${DRY_RUN}" == "1" ]]; then
|
||||
CMD+=(--dry-run)
|
||||
fi
|
||||
|
||||
if [[ "${SKIP_EXISTING}" == "1" ]]; then
|
||||
CMD+=(--skip-existing)
|
||||
fi
|
||||
|
||||
if [[ -n "${DECODE_ALL_ISSUE_IDS}" ]]; then
|
||||
# shellcheck disable=SC2206
|
||||
ISSUE_ARR=(${DECODE_ALL_ISSUE_IDS})
|
||||
for issue_id in "${ISSUE_ARR[@]}"; do
|
||||
CMD+=(--decode-all-issue-id "${issue_id}")
|
||||
done
|
||||
fi
|
||||
|
||||
CMD+=("$@")
|
||||
"${CMD[@]}"
|
||||
1111
tools/feishu_project/dongying_g1q3_issue_list.json
Executable file
1111
tools/feishu_project/dongying_g1q3_issue_list.json
Executable file
File diff suppressed because it is too large
Load Diff
805
tools/feishu_project/download_issue_data.py
Executable file
805
tools/feishu_project/download_issue_data.py
Executable file
@@ -0,0 +1,805 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Download issue data referenced by the Feishu issue export JSON."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
from collections import OrderedDict
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Iterable
|
||||
|
||||
FILE = Path(__file__).resolve()
|
||||
ROOT = FILE.parents[2]
|
||||
if str(ROOT) not in sys.path:
|
||||
sys.path.append(str(ROOT))
|
||||
|
||||
from tools.feishu_project.case_calib_recovery import recover_camera4_json
|
||||
|
||||
|
||||
PDCL_REF_RE = re.compile(r"ADAS_[^:/\\\s]+::[^/\\\s]*")
|
||||
MDI_RAW_REF_ARG_RE = re.compile(r"(?:^|\s)mdi\s+raw\b.*?(?:^|\s)-r\s+([^\s]+)")
|
||||
STANDARD_PATH_SPLIT_RE = re.compile(r"[,,\n;;]+")
|
||||
SHARED_CALIB_REL = Path("test_data") / "calibs" / "camera4.json"
|
||||
PLACEHOLDER_TEXTS = {"待填", "待补充", "none", "null", "待提供"}
|
||||
NETWORK_SHARE_PREFIX_MAPPINGS = (
|
||||
("//hfs.minieye.tech/project-D4Q2", "/mnt/D4Q2"),
|
||||
("//192.168.2.122/project-D4Q2", "/mnt/D4Q2"),
|
||||
("//hfs.minieye.tech/project-G1M3", "/mnt/G1M3"),
|
||||
("//192.168.2.122/project-G1M3", "/mnt/G1M3"),
|
||||
("//hfs.minieye.tech/G1M3", "/mnt/G1M3"),
|
||||
("//192.168.2.122/G1M3", "/mnt/G1M3"),
|
||||
)
|
||||
|
||||
|
||||
@dataclass
|
||||
class ActionResult:
|
||||
issue_id: int
|
||||
issue_name: str
|
||||
source_field: str
|
||||
source_kind: str
|
||||
raw_value: str | None
|
||||
normalized_ref: str | None
|
||||
output_dir: str
|
||||
status: str
|
||||
detail: str
|
||||
resolved_source_path: str | None = None
|
||||
command: list[str] | None = None
|
||||
candidate_paths: list[str] | None = None
|
||||
selected_subpath: str | None = None
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
return {
|
||||
"issue_id": self.issue_id,
|
||||
"issue_name": self.issue_name,
|
||||
"source_field": self.source_field,
|
||||
"source_kind": self.source_kind,
|
||||
"raw_value": self.raw_value,
|
||||
"normalized_ref": self.normalized_ref,
|
||||
"output_dir": self.output_dir,
|
||||
"status": self.status,
|
||||
"detail": self.detail,
|
||||
"resolved_source_path": self.resolved_source_path,
|
||||
"command": self.command,
|
||||
"candidate_paths": self.candidate_paths,
|
||||
"selected_subpath": self.selected_subpath,
|
||||
}
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class PDCLDownloadRequest:
|
||||
normalized_ref: str
|
||||
selected_subpath: str | None = None
|
||||
raw_token: str | None = None
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Download or stage issue data from a Feishu issue export JSON."
|
||||
)
|
||||
parser.add_argument(
|
||||
"--input-json",
|
||||
default="tools/feishu_project/dongying_g1q3_issue_list.json",
|
||||
help="Path to the issue export JSON.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--output-root",
|
||||
required=True,
|
||||
help="Directory where downloaded or copied data should be stored.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--manifest-path",
|
||||
default=None,
|
||||
help="Optional explicit path for the execution manifest JSON.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--issue-id",
|
||||
action="append",
|
||||
dest="issue_ids",
|
||||
type=int,
|
||||
help="Optional issue id filter. Can be repeated.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--dry-run",
|
||||
action="store_true",
|
||||
help="Plan actions without running mdi or copying files.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--skip-mdi",
|
||||
action="store_true",
|
||||
help="Skip PDCL/MDI downloads and only process standard paths.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--skip-copy",
|
||||
action="store_true",
|
||||
help="Skip standard-path copies and only process PDCL/MDI downloads.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--only-redownload-affected-cases",
|
||||
action="store_true",
|
||||
help=(
|
||||
"Only re-copy standard-path cases affected by the historical sigmastar.1/camera4.bin "
|
||||
"copy bug. This mode skips PDCL/MDI downloads and replaces stale copied targets."
|
||||
),
|
||||
)
|
||||
parser.add_argument(
|
||||
"--skip-calib-recovery",
|
||||
action="store_true",
|
||||
help=(
|
||||
"Skip recovering camera4.json from camera_config_folder.bin or mcap attachments "
|
||||
"after standard-path copies."
|
||||
),
|
||||
)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def load_issue_items(path: Path) -> list[dict]:
|
||||
payload = json.loads(path.read_text(encoding="utf-8"))
|
||||
return payload["items"]
|
||||
|
||||
|
||||
def ensure_dir(path: Path, dry_run: bool) -> None:
|
||||
if dry_run:
|
||||
return
|
||||
path.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
|
||||
def log_progress(message: str) -> None:
|
||||
timestamp = datetime.now().astimezone().strftime("%Y-%m-%d %H:%M:%S")
|
||||
print(f"[download_issue_data {timestamp}] {message}", flush=True)
|
||||
|
||||
|
||||
def compact_text(value: object, max_len: int = 96) -> str:
|
||||
text = "" if value is None else str(value).strip()
|
||||
text = re.sub(r"\s+", " ", text)
|
||||
if len(text) <= max_len:
|
||||
return text
|
||||
return f"{text[: max_len - 3]}..."
|
||||
|
||||
|
||||
def summarize_issue_results(results: list[ActionResult]) -> str:
|
||||
if not results:
|
||||
return "no actions"
|
||||
summary: dict[str, int] = {}
|
||||
for result in results:
|
||||
summary[result.status] = summary.get(result.status, 0) + 1
|
||||
return ", ".join(f"{status}={summary[status]}" for status in sorted(summary))
|
||||
|
||||
|
||||
def normalize_issue_dirname(issue_id: int) -> str:
|
||||
return f"issue_{issue_id}"
|
||||
|
||||
|
||||
def iter_issue_fields(item: dict) -> Iterable[tuple[str, object]]:
|
||||
yield "问题数据地址", item.get("问题数据地址")
|
||||
yield "问题数据地址_PDCL", item.get("问题数据地址_PDCL")
|
||||
|
||||
|
||||
def _normalize_pdcl_selected_subpath(raw_subpath: str) -> str | None:
|
||||
cleaned = raw_subpath.strip().strip("/")
|
||||
if not cleaned:
|
||||
return None
|
||||
candidate = Path(cleaned)
|
||||
if candidate.is_absolute():
|
||||
return None
|
||||
if any(part in {"", ".", ".."} for part in candidate.parts):
|
||||
return None
|
||||
return str(candidate)
|
||||
|
||||
|
||||
def _build_pdcl_request_from_token(token: str) -> PDCLDownloadRequest | None:
|
||||
stripped = token.strip().strip("\"'`")
|
||||
match = PDCL_REF_RE.match(stripped)
|
||||
if match is None:
|
||||
return None
|
||||
normalized_ref = match.group(0)
|
||||
suffix = stripped[match.end():]
|
||||
selected_subpath = None
|
||||
if suffix.startswith("/"):
|
||||
raw_subpath = suffix[1:]
|
||||
if raw_subpath and not raw_subpath.startswith("ADAS_"):
|
||||
selected_subpath = _normalize_pdcl_selected_subpath(raw_subpath)
|
||||
return PDCLDownloadRequest(
|
||||
normalized_ref=normalized_ref,
|
||||
selected_subpath=selected_subpath,
|
||||
raw_token=stripped,
|
||||
)
|
||||
|
||||
|
||||
def extract_pdcl_requests(raw_value: object) -> list[PDCLDownloadRequest]:
|
||||
if raw_value is None:
|
||||
return []
|
||||
text = str(raw_value).strip()
|
||||
if not text:
|
||||
return []
|
||||
requests: list[PDCLDownloadRequest] = []
|
||||
|
||||
for segment in (part.strip() for part in STANDARD_PATH_SPLIT_RE.split(text)):
|
||||
if not segment:
|
||||
continue
|
||||
|
||||
mdi_match = MDI_RAW_REF_ARG_RE.search(segment)
|
||||
if mdi_match is not None:
|
||||
request = _build_pdcl_request_from_token(mdi_match.group(1))
|
||||
if request is not None:
|
||||
requests.append(request)
|
||||
continue
|
||||
|
||||
search_pos = 0
|
||||
while True:
|
||||
match = PDCL_REF_RE.search(segment, search_pos)
|
||||
if match is None:
|
||||
break
|
||||
normalized_ref = match.group(0)
|
||||
suffix = segment[match.end():]
|
||||
selected_subpath = None
|
||||
if suffix.startswith("/") and not suffix[1:].startswith("ADAS_"):
|
||||
selected_subpath = _normalize_pdcl_selected_subpath(suffix[1:])
|
||||
requests.append(
|
||||
PDCLDownloadRequest(
|
||||
normalized_ref=normalized_ref,
|
||||
selected_subpath=selected_subpath,
|
||||
raw_token=segment,
|
||||
)
|
||||
)
|
||||
break
|
||||
|
||||
requests.append(
|
||||
PDCLDownloadRequest(
|
||||
normalized_ref=normalized_ref,
|
||||
selected_subpath=None,
|
||||
raw_token=normalized_ref,
|
||||
)
|
||||
)
|
||||
search_pos = match.end()
|
||||
|
||||
deduped: list[PDCLDownloadRequest] = []
|
||||
seen = set()
|
||||
for request in requests:
|
||||
key = (request.normalized_ref, request.selected_subpath)
|
||||
if key in seen:
|
||||
continue
|
||||
seen.add(key)
|
||||
deduped.append(request)
|
||||
return deduped
|
||||
|
||||
|
||||
def extract_standard_paths(raw_value: object) -> list[str]:
|
||||
if raw_value is None:
|
||||
return []
|
||||
text = str(raw_value).strip()
|
||||
if not text:
|
||||
return []
|
||||
if text.lower() in PLACEHOLDER_TEXTS:
|
||||
return []
|
||||
if extract_pdcl_requests(text):
|
||||
return []
|
||||
if "/" not in text and "\\" not in text:
|
||||
return []
|
||||
parts = [part.strip() for part in STANDARD_PATH_SPLIT_RE.split(text)]
|
||||
return [part for part in parts if part]
|
||||
|
||||
|
||||
def normalize_standard_source_path(path: Path) -> Path:
|
||||
normalized = path
|
||||
if normalized.name == "camera4.bin" and normalized.parent.name == "sigmastar.1":
|
||||
return normalized.parent.parent
|
||||
if normalized.name == "sigmastar.1":
|
||||
return normalized.parent
|
||||
return normalized
|
||||
|
||||
|
||||
def is_affected_standard_path(raw_path: str) -> bool:
|
||||
raw_path_obj = Path(raw_path.strip())
|
||||
return normalize_standard_source_path(raw_path_obj) != raw_path_obj
|
||||
|
||||
|
||||
def normalize_share_path_separators(path_str: str) -> str:
|
||||
normalized = path_str.strip().replace("\\", "/")
|
||||
normalized = re.sub(r"/{3,}", "//", normalized)
|
||||
return normalized
|
||||
|
||||
|
||||
def rewrite_network_share_path(path_str: str) -> str | None:
|
||||
normalized = normalize_share_path_separators(path_str)
|
||||
for prefix_src, prefix_dst in NETWORK_SHARE_PREFIX_MAPPINGS:
|
||||
if normalized.startswith(prefix_src):
|
||||
return f"{prefix_dst}{normalized[len(prefix_src):]}"
|
||||
return None
|
||||
|
||||
|
||||
def build_path_candidates(raw_path: str) -> list[Path]:
|
||||
candidates: list[str] = [raw_path]
|
||||
|
||||
normalized_share = normalize_share_path_separators(raw_path)
|
||||
if normalized_share != raw_path:
|
||||
candidates.append(normalized_share)
|
||||
|
||||
network_share_rewritten = rewrite_network_share_path(raw_path)
|
||||
if network_share_rewritten:
|
||||
candidates.append(network_share_rewritten)
|
||||
|
||||
for needle in ("hfs/project-G1M3", "project-G1M3"):
|
||||
for candidate in list(candidates):
|
||||
if needle in candidate:
|
||||
candidates.append(candidate.replace(needle, "G1M3"))
|
||||
|
||||
normalized_candidates = [normalize_standard_source_path(Path(candidate)) for candidate in candidates]
|
||||
unique_candidates = list(OrderedDict.fromkeys(str(candidate) for candidate in normalized_candidates))
|
||||
return [Path(candidate) for candidate in unique_candidates]
|
||||
|
||||
|
||||
def resolve_existing_path(raw_path: str) -> tuple[Path | None, list[Path]]:
|
||||
candidates = build_path_candidates(raw_path)
|
||||
for candidate in candidates:
|
||||
if candidate.exists():
|
||||
return candidate, candidates
|
||||
return None, candidates
|
||||
|
||||
|
||||
def remove_existing_target(path: Path) -> None:
|
||||
if path.is_dir():
|
||||
shutil.rmtree(path)
|
||||
else:
|
||||
path.unlink()
|
||||
|
||||
|
||||
def copy_source_path(
|
||||
source_path: Path,
|
||||
output_dir: Path,
|
||||
dry_run: bool,
|
||||
replace_existing: bool = False,
|
||||
legacy_target_names: Iterable[str] | None = None,
|
||||
) -> tuple[str, str]:
|
||||
target = output_dir / source_path.name
|
||||
cleanup_targets: list[Path] = [target]
|
||||
if legacy_target_names:
|
||||
for target_name in legacy_target_names:
|
||||
cleanup_targets.append(output_dir / target_name)
|
||||
|
||||
unique_cleanup_targets = list(OrderedDict.fromkeys(str(path) for path in cleanup_targets))
|
||||
cleanup_paths = [Path(path) for path in unique_cleanup_targets]
|
||||
existing_targets = [path for path in cleanup_paths if path.exists()]
|
||||
|
||||
if existing_targets and not replace_existing:
|
||||
return "exists", f"target already exists: {target}"
|
||||
|
||||
if dry_run:
|
||||
if existing_targets and replace_existing:
|
||||
existing_str = ", ".join(str(path) for path in existing_targets)
|
||||
return "planned_redownload", f"would replace {existing_str} with {source_path} -> {target}"
|
||||
return "planned", f"would copy {source_path} -> {target}"
|
||||
|
||||
ensure_dir(output_dir, dry_run=False)
|
||||
if replace_existing:
|
||||
for existing_target in existing_targets:
|
||||
remove_existing_target(existing_target)
|
||||
|
||||
if source_path.is_dir():
|
||||
shutil.copytree(source_path, target)
|
||||
else:
|
||||
shutil.copy2(source_path, target)
|
||||
if existing_targets and replace_existing:
|
||||
replaced = ", ".join(str(path) for path in existing_targets)
|
||||
return "redownloaded", f"replaced {replaced} with {target}"
|
||||
return "copied", f"copied to {target}"
|
||||
|
||||
|
||||
def build_copied_target_root(output_dir: Path, source_path: Path) -> Path:
|
||||
return output_dir / source_path.name
|
||||
|
||||
|
||||
def find_shared_test_data_dir(source_path: Path, max_parent_levels: int = 4) -> Path | None:
|
||||
if (source_path / SHARED_CALIB_REL).is_file():
|
||||
return None
|
||||
|
||||
current = source_path.parent
|
||||
for _ in range(max_parent_levels):
|
||||
candidate = current / "test_data"
|
||||
if (candidate / "calibs" / "camera4.json").is_file():
|
||||
return candidate
|
||||
if current.parent == current:
|
||||
break
|
||||
current = current.parent
|
||||
return None
|
||||
|
||||
|
||||
def sync_shared_test_data(
|
||||
source_path: Path,
|
||||
target_root: Path,
|
||||
dry_run: bool,
|
||||
) -> tuple[str | None, str | None, str | None]:
|
||||
shared_test_data_dir = find_shared_test_data_dir(source_path)
|
||||
if shared_test_data_dir is None:
|
||||
return None, None, None
|
||||
|
||||
target_shared_test_data_dir = target_root / "test_data"
|
||||
target_shared_calib = target_shared_test_data_dir / "calibs" / "camera4.json"
|
||||
if target_shared_calib.is_file():
|
||||
return None, None, None
|
||||
|
||||
if dry_run:
|
||||
return (
|
||||
"planned_shared_calib_sync",
|
||||
f"would copy shared test_data {shared_test_data_dir} -> {target_shared_test_data_dir}",
|
||||
str(shared_test_data_dir),
|
||||
)
|
||||
|
||||
shutil.copytree(shared_test_data_dir, target_shared_test_data_dir, dirs_exist_ok=True)
|
||||
return (
|
||||
"synced_shared_calib",
|
||||
f"copied shared test_data {shared_test_data_dir} -> {target_shared_test_data_dir}",
|
||||
str(shared_test_data_dir),
|
||||
)
|
||||
|
||||
|
||||
def recover_target_root_calib(
|
||||
source_root: Path,
|
||||
target_root: Path,
|
||||
dry_run: bool,
|
||||
) -> tuple[str, str, str | None]:
|
||||
recovery = recover_camera4_json(
|
||||
source_root=source_root,
|
||||
target_root=target_root,
|
||||
dry_run=dry_run,
|
||||
)
|
||||
return recovery.status, recovery.detail, None if recovery.source_path is None else str(recovery.source_path)
|
||||
|
||||
|
||||
def _expected_pdcl_root_dir(output_dir: Path, ref: str) -> Path:
|
||||
if "::" not in ref:
|
||||
raise ValueError(f"Unexpected PDCL ref without '::': {ref}")
|
||||
return output_dir / ref.split("::", 1)[1]
|
||||
|
||||
|
||||
def _prune_pdcl_download_to_selected_subpath(
|
||||
root_dir: Path,
|
||||
selected_subpath: str,
|
||||
dry_run: bool,
|
||||
) -> tuple[str, str]:
|
||||
selected_rel = Path(selected_subpath)
|
||||
selected_path = root_dir / selected_rel
|
||||
if dry_run:
|
||||
return (
|
||||
"planned_selected_subpath",
|
||||
f"would keep {selected_path} and shared test_data under {root_dir}",
|
||||
)
|
||||
|
||||
if not selected_path.exists():
|
||||
return (
|
||||
"failed_selected_subpath_missing",
|
||||
f"selected subpath not found after mdi download: {selected_path}",
|
||||
)
|
||||
|
||||
keep_names = {selected_rel.parts[0], "test_data"}
|
||||
removed_children: list[str] = []
|
||||
|
||||
for child in root_dir.iterdir():
|
||||
if child.name in keep_names:
|
||||
continue
|
||||
removed_children.append(child.name)
|
||||
remove_existing_target(child)
|
||||
|
||||
detail = f"kept selected subpath {selected_path}"
|
||||
if removed_children:
|
||||
detail += f"; removed siblings: {', '.join(sorted(removed_children))}"
|
||||
return "downloaded_selected_subpath", detail
|
||||
|
||||
|
||||
def run_mdi_download(request: PDCLDownloadRequest, output_dir: Path, dry_run: bool) -> tuple[str, str, list[str]]:
|
||||
command = ["mdi", "raw", "-r", request.normalized_ref, "-s", str(output_dir)]
|
||||
if dry_run:
|
||||
if request.selected_subpath:
|
||||
root_dir = _expected_pdcl_root_dir(output_dir, request.normalized_ref)
|
||||
status, detail = _prune_pdcl_download_to_selected_subpath(root_dir, request.selected_subpath, dry_run=True)
|
||||
return status, f"would run {' '.join(command)}; {detail}", command
|
||||
return "planned", f"would run {' '.join(command)}", command
|
||||
|
||||
ensure_dir(output_dir, dry_run=False)
|
||||
completed = subprocess.run(
|
||||
command,
|
||||
check=False,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
encoding="utf-8",
|
||||
)
|
||||
if completed.returncode == 0:
|
||||
base_detail = completed.stdout.strip() or "mdi raw completed"
|
||||
if request.selected_subpath:
|
||||
root_dir = _expected_pdcl_root_dir(output_dir, request.normalized_ref)
|
||||
status, detail = _prune_pdcl_download_to_selected_subpath(root_dir, request.selected_subpath, dry_run=False)
|
||||
return status, f"{base_detail}\n{detail}", command
|
||||
return "downloaded", base_detail, command
|
||||
|
||||
detail = completed.stderr.strip() or completed.stdout.strip() or "mdi raw failed"
|
||||
return "failed", detail, command
|
||||
|
||||
|
||||
def process_issue(
|
||||
item: dict,
|
||||
output_root: Path,
|
||||
dry_run: bool,
|
||||
skip_mdi: bool,
|
||||
skip_copy: bool,
|
||||
only_redownload_affected_cases: bool,
|
||||
skip_calib_recovery: bool,
|
||||
) -> list[ActionResult]:
|
||||
issue_id = int(item["id"])
|
||||
issue_name = str(item["name"])
|
||||
issue_dir = output_root / normalize_issue_dirname(issue_id)
|
||||
|
||||
results: list[ActionResult] = []
|
||||
seen_pdcl_requests: set[tuple[str, str | None]] = set()
|
||||
seen_paths: set[str] = set()
|
||||
pdcl_index = 0
|
||||
path_index = 0
|
||||
|
||||
for field_name, raw_value in iter_issue_fields(item):
|
||||
if not skip_mdi and not only_redownload_affected_cases:
|
||||
for request in extract_pdcl_requests(raw_value):
|
||||
request_key = (request.normalized_ref, request.selected_subpath)
|
||||
if request_key in seen_pdcl_requests:
|
||||
results.append(
|
||||
ActionResult(
|
||||
issue_id=issue_id,
|
||||
issue_name=issue_name,
|
||||
source_field=field_name,
|
||||
source_kind="pdcl_mdi_download",
|
||||
raw_value=None if raw_value is None else str(raw_value),
|
||||
normalized_ref=request.normalized_ref,
|
||||
output_dir=str(issue_dir),
|
||||
status="skipped_duplicate",
|
||||
detail=(
|
||||
f"duplicate PDCL ref: {request.normalized_ref}"
|
||||
if request.selected_subpath is None
|
||||
else f"duplicate PDCL ref+subpath: {request.normalized_ref} / {request.selected_subpath}"
|
||||
),
|
||||
selected_subpath=request.selected_subpath,
|
||||
)
|
||||
)
|
||||
continue
|
||||
|
||||
seen_pdcl_requests.add(request_key)
|
||||
pdcl_index += 1
|
||||
download_dir = issue_dir / f"pdcl_{pdcl_index:02d}"
|
||||
request_desc = request.normalized_ref
|
||||
if request.selected_subpath:
|
||||
request_desc += f"/{request.selected_subpath}"
|
||||
log_progress(
|
||||
f"issue_{issue_id} [download] pdcl_{pdcl_index:02d}: {compact_text(request_desc)}"
|
||||
)
|
||||
status, detail, command = run_mdi_download(request, download_dir, dry_run=dry_run)
|
||||
results.append(
|
||||
ActionResult(
|
||||
issue_id=issue_id,
|
||||
issue_name=issue_name,
|
||||
source_field=field_name,
|
||||
source_kind="pdcl_mdi_download",
|
||||
raw_value=None if raw_value is None else str(raw_value),
|
||||
normalized_ref=request.normalized_ref,
|
||||
output_dir=str(download_dir),
|
||||
status=status,
|
||||
detail=detail,
|
||||
command=command,
|
||||
selected_subpath=request.selected_subpath,
|
||||
)
|
||||
)
|
||||
|
||||
if skip_copy:
|
||||
continue
|
||||
|
||||
for raw_path in extract_standard_paths(raw_value):
|
||||
if raw_path in seen_paths:
|
||||
results.append(
|
||||
ActionResult(
|
||||
issue_id=issue_id,
|
||||
issue_name=issue_name,
|
||||
source_field=field_name,
|
||||
source_kind="standard_path",
|
||||
raw_value=raw_path,
|
||||
normalized_ref=None,
|
||||
output_dir=str(issue_dir),
|
||||
status="skipped_duplicate",
|
||||
detail=f"duplicate standard path: {raw_path}",
|
||||
)
|
||||
)
|
||||
continue
|
||||
|
||||
seen_paths.add(raw_path)
|
||||
path_index += 1
|
||||
copy_dir = issue_dir / f"path_{path_index:02d}"
|
||||
affected_standard_path = is_affected_standard_path(raw_path)
|
||||
if only_redownload_affected_cases and not affected_standard_path:
|
||||
continue
|
||||
|
||||
log_progress(
|
||||
f"issue_{issue_id} [download] path_{path_index:02d}: {compact_text(raw_path)}"
|
||||
)
|
||||
resolved_source_path, candidates = resolve_existing_path(raw_path)
|
||||
if resolved_source_path is None:
|
||||
results.append(
|
||||
ActionResult(
|
||||
issue_id=issue_id,
|
||||
issue_name=issue_name,
|
||||
source_field=field_name,
|
||||
source_kind="standard_path",
|
||||
raw_value=raw_path,
|
||||
normalized_ref=None,
|
||||
output_dir=str(copy_dir),
|
||||
status="skipped_missing",
|
||||
detail="source path not found after rewrite attempts",
|
||||
candidate_paths=[str(candidate) for candidate in candidates],
|
||||
)
|
||||
)
|
||||
continue
|
||||
|
||||
legacy_target_names = []
|
||||
if affected_standard_path:
|
||||
legacy_target_names.append(Path(raw_path.strip()).name)
|
||||
|
||||
status, detail = copy_source_path(
|
||||
resolved_source_path,
|
||||
copy_dir,
|
||||
dry_run=dry_run,
|
||||
replace_existing=only_redownload_affected_cases and affected_standard_path,
|
||||
legacy_target_names=legacy_target_names,
|
||||
)
|
||||
target_root = build_copied_target_root(copy_dir, resolved_source_path)
|
||||
results.append(
|
||||
ActionResult(
|
||||
issue_id=issue_id,
|
||||
issue_name=issue_name,
|
||||
source_field=field_name,
|
||||
source_kind="standard_path",
|
||||
raw_value=raw_path,
|
||||
normalized_ref=None,
|
||||
output_dir=str(copy_dir),
|
||||
status=status,
|
||||
detail=detail,
|
||||
resolved_source_path=str(resolved_source_path),
|
||||
candidate_paths=[str(candidate) for candidate in candidates],
|
||||
)
|
||||
)
|
||||
|
||||
sync_status, sync_detail, shared_source_dir = sync_shared_test_data(
|
||||
resolved_source_path,
|
||||
target_root,
|
||||
dry_run=dry_run,
|
||||
)
|
||||
if sync_status is not None:
|
||||
results.append(
|
||||
ActionResult(
|
||||
issue_id=issue_id,
|
||||
issue_name=issue_name,
|
||||
source_field=field_name,
|
||||
source_kind="shared_test_data",
|
||||
raw_value=raw_path,
|
||||
normalized_ref=None,
|
||||
output_dir=str(target_root / "test_data"),
|
||||
status=sync_status,
|
||||
detail=sync_detail or "",
|
||||
resolved_source_path=shared_source_dir,
|
||||
candidate_paths=[str(candidate) for candidate in candidates],
|
||||
)
|
||||
)
|
||||
|
||||
if not skip_calib_recovery:
|
||||
calib_status, calib_detail, calib_source_path = recover_target_root_calib(
|
||||
source_root=resolved_source_path,
|
||||
target_root=target_root,
|
||||
dry_run=dry_run,
|
||||
)
|
||||
results.append(
|
||||
ActionResult(
|
||||
issue_id=issue_id,
|
||||
issue_name=issue_name,
|
||||
source_field=field_name,
|
||||
source_kind="case_calib_recovery",
|
||||
raw_value=raw_path,
|
||||
normalized_ref=None,
|
||||
output_dir=str(target_root / "test_data" / "calibs"),
|
||||
status=calib_status,
|
||||
detail=calib_detail,
|
||||
resolved_source_path=calib_source_path,
|
||||
candidate_paths=[str(candidate) for candidate in candidates],
|
||||
)
|
||||
)
|
||||
|
||||
return results
|
||||
|
||||
|
||||
def build_manifest(
|
||||
args: argparse.Namespace,
|
||||
input_json: Path,
|
||||
output_root: Path,
|
||||
action_results: list[ActionResult],
|
||||
) -> dict:
|
||||
summary: dict[str, int] = {}
|
||||
for action in action_results:
|
||||
summary[action.status] = summary.get(action.status, 0) + 1
|
||||
|
||||
return {
|
||||
"generated_at": datetime.now().astimezone().isoformat(timespec="seconds"),
|
||||
"input_json": str(input_json),
|
||||
"output_root": str(output_root),
|
||||
"dry_run": args.dry_run,
|
||||
"skip_mdi": args.skip_mdi,
|
||||
"skip_copy": args.skip_copy,
|
||||
"skip_calib_recovery": args.skip_calib_recovery,
|
||||
"only_redownload_affected_cases": args.only_redownload_affected_cases,
|
||||
"issue_filter": args.issue_ids or [],
|
||||
"summary": summary,
|
||||
"actions": [action.to_dict() for action in action_results],
|
||||
}
|
||||
|
||||
|
||||
def print_summary(manifest: dict) -> None:
|
||||
print(f"input_json: {manifest['input_json']}")
|
||||
print(f"output_root: {manifest['output_root']}")
|
||||
print(f"dry_run: {manifest['dry_run']}")
|
||||
for status, count in sorted(manifest["summary"].items()):
|
||||
print(f"{status}: {count}")
|
||||
|
||||
|
||||
def main() -> int:
|
||||
args = parse_args()
|
||||
input_json = Path(args.input_json).resolve()
|
||||
output_root = Path(args.output_root).resolve()
|
||||
manifest_path = (
|
||||
Path(args.manifest_path).resolve()
|
||||
if args.manifest_path
|
||||
else output_root / "download_manifest.json"
|
||||
)
|
||||
|
||||
items = load_issue_items(input_json)
|
||||
if args.issue_ids:
|
||||
issue_filter = set(args.issue_ids)
|
||||
items = [item for item in items if int(item["id"]) in issue_filter]
|
||||
|
||||
ensure_dir(output_root, dry_run=args.dry_run)
|
||||
|
||||
action_results: list[ActionResult] = []
|
||||
total_items = len(items)
|
||||
log_progress(f"issues_to_process: {total_items}")
|
||||
for index, item in enumerate(items, start=1):
|
||||
issue_id = int(item["id"])
|
||||
issue_name = compact_text(item.get("name"), max_len=64)
|
||||
log_progress(f"[{index}/{total_items}] issue_{issue_id} start: {issue_name}")
|
||||
issue_results = process_issue(
|
||||
item=item,
|
||||
output_root=output_root,
|
||||
dry_run=args.dry_run,
|
||||
skip_mdi=args.skip_mdi,
|
||||
skip_copy=args.skip_copy,
|
||||
only_redownload_affected_cases=args.only_redownload_affected_cases,
|
||||
skip_calib_recovery=args.skip_calib_recovery,
|
||||
)
|
||||
action_results.extend(issue_results)
|
||||
log_progress(
|
||||
f"[{index}/{total_items}] issue_{issue_id} done: {summarize_issue_results(issue_results)}"
|
||||
)
|
||||
|
||||
manifest = build_manifest(args, input_json, output_root, action_results)
|
||||
ensure_dir(manifest_path.parent, dry_run=args.dry_run)
|
||||
if not args.dry_run:
|
||||
manifest_path.write_text(
|
||||
json.dumps(manifest, ensure_ascii=False, indent=2) + "\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
print_summary(manifest)
|
||||
if args.dry_run:
|
||||
print(f"manifest (not written in dry-run): {manifest_path}")
|
||||
else:
|
||||
print(f"manifest: {manifest_path}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
38
tools/feishu_project/download_issue_data.sh
Executable file
38
tools/feishu_project/download_issue_data.sh
Executable file
@@ -0,0 +1,38 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
|
||||
PROJECT_ROOT=$(cd "${SCRIPT_DIR}/../.." && pwd)
|
||||
|
||||
SYNC_ROOT=${SYNC_ROOT:-/data1/dongying/Mono3d/G1Q3/feishu_project}
|
||||
DEFAULT_INPUT_JSON="${SYNC_ROOT}/exports/dongying_g1q3_issue_list.json"
|
||||
FALLBACK_INPUT_JSON="${PROJECT_ROOT}/tools/feishu_project/dongying_g1q3_issue_list.json"
|
||||
INPUT_JSON=${INPUT_JSON:-"${DEFAULT_INPUT_JSON}"}
|
||||
OUTPUT_ROOT=${OUTPUT_ROOT:-/data1/dongying/Mono3d/G1Q3/feishu_project/downloaded_issue_data}
|
||||
MANIFEST_PATH=${MANIFEST_PATH:-"${OUTPUT_ROOT}/download_manifest.json"}
|
||||
PYTHON_BIN=${PYTHON_BIN:-/deeplearning_team/ydong/dongying/miniconda/envs/dev/bin/python}
|
||||
|
||||
if [[ ! -f "${INPUT_JSON}" && -f "${FALLBACK_INPUT_JSON}" ]]; then
|
||||
INPUT_JSON="${FALLBACK_INPUT_JSON}"
|
||||
fi
|
||||
|
||||
EXTRA_ARGS=()
|
||||
if [[ "${DRY_RUN:-0}" == "1" ]]; then
|
||||
EXTRA_ARGS+=("--dry-run")
|
||||
fi
|
||||
if [[ "${SKIP_MDI:-0}" == "1" ]]; then
|
||||
EXTRA_ARGS+=("--skip-mdi")
|
||||
fi
|
||||
if [[ "${SKIP_COPY:-0}" == "1" ]]; then
|
||||
EXTRA_ARGS+=("--skip-copy")
|
||||
fi
|
||||
if [[ "${ONLY_REDOWNLOAD_AFFECTED_CASES:-0}" == "1" ]]; then
|
||||
EXTRA_ARGS+=("--only-redownload-affected-cases")
|
||||
fi
|
||||
|
||||
"${PYTHON_BIN}" "${PROJECT_ROOT}/tools/feishu_project/download_issue_data.py" \
|
||||
--input-json "${INPUT_JSON}" \
|
||||
--output-root "${OUTPUT_ROOT}" \
|
||||
--manifest-path "${MANIFEST_PATH}" \
|
||||
"${EXTRA_ARGS[@]}" \
|
||||
"$@"
|
||||
209
tools/feishu_project/export_feishu_view_issues.py
Executable file
209
tools/feishu_project/export_feishu_view_issues.py
Executable file
@@ -0,0 +1,209 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Export Feishu view issues and enrich them with selected detail fields."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
DEFAULT_EXTRA_FIELDS = [
|
||||
"缺陷标签池",
|
||||
"问题数据地址",
|
||||
"问题数据地址_PDCL",
|
||||
"问题发生frameid",
|
||||
]
|
||||
|
||||
|
||||
def run_fp(args: list[str]) -> dict:
|
||||
result = subprocess.run(
|
||||
["fp", *args],
|
||||
check=False,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
encoding="utf-8",
|
||||
)
|
||||
if result.returncode != 0:
|
||||
stderr = result.stderr.strip()
|
||||
stdout = result.stdout.strip()
|
||||
detail = stderr or stdout or "unknown error"
|
||||
raise RuntimeError(f"fp command failed: {' '.join(args)}\n{detail}")
|
||||
try:
|
||||
return json.loads(result.stdout)
|
||||
except json.JSONDecodeError as exc:
|
||||
raise RuntimeError(
|
||||
f"fp returned non-JSON output for command: {' '.join(args)}"
|
||||
) from exc
|
||||
|
||||
|
||||
def fetch_view(project_key: str, user_key: str, view_name: str, work_item_type: str) -> dict | None:
|
||||
payload = run_fp(
|
||||
[
|
||||
"view",
|
||||
"list",
|
||||
"-o",
|
||||
"json",
|
||||
"-p",
|
||||
project_key,
|
||||
"-u",
|
||||
user_key,
|
||||
"-t",
|
||||
work_item_type,
|
||||
"--name",
|
||||
view_name,
|
||||
]
|
||||
)
|
||||
views = payload.get("data", [])
|
||||
if not views:
|
||||
return None
|
||||
return views[0]
|
||||
|
||||
|
||||
def fetch_items(project_key: str, user_key: str, view_name: str, work_item_type: str) -> dict:
|
||||
return run_fp(
|
||||
[
|
||||
"workitem",
|
||||
"list",
|
||||
"-o",
|
||||
"json",
|
||||
"--all",
|
||||
"-p",
|
||||
project_key,
|
||||
"-u",
|
||||
user_key,
|
||||
"-t",
|
||||
work_item_type,
|
||||
"--view",
|
||||
view_name,
|
||||
]
|
||||
)
|
||||
|
||||
|
||||
def fetch_item_detail(project_key: str, user_key: str, item_id: int, work_item_type: str) -> dict:
|
||||
payload = run_fp(
|
||||
[
|
||||
"workitem",
|
||||
"get",
|
||||
str(item_id),
|
||||
"-o",
|
||||
"json",
|
||||
"-p",
|
||||
project_key,
|
||||
"-u",
|
||||
user_key,
|
||||
"-t",
|
||||
work_item_type,
|
||||
]
|
||||
)
|
||||
return payload["data"]
|
||||
|
||||
|
||||
def enrich_items(
|
||||
base_items: list[dict],
|
||||
project_key: str,
|
||||
user_key: str,
|
||||
work_item_type: str,
|
||||
extra_fields: list[str],
|
||||
) -> list[dict]:
|
||||
enriched = []
|
||||
for item in base_items:
|
||||
detail = fetch_item_detail(project_key, user_key, item["id"], work_item_type)
|
||||
detail_fields = detail.get("fields", {})
|
||||
merged = dict(item)
|
||||
for field_name in extra_fields:
|
||||
merged[field_name] = detail_fields.get(field_name)
|
||||
enriched.append(merged)
|
||||
return enriched
|
||||
|
||||
|
||||
def build_output(
|
||||
project_key: str,
|
||||
user_key: str,
|
||||
view_name: str,
|
||||
view: dict | None,
|
||||
items_payload: dict,
|
||||
extra_fields: list[str],
|
||||
work_item_type: str,
|
||||
) -> dict:
|
||||
return {
|
||||
"exported_at": datetime.now().astimezone().isoformat(timespec="seconds"),
|
||||
"source": {
|
||||
"project_key": project_key,
|
||||
"user_key": user_key,
|
||||
"view_name": view_name,
|
||||
"view_id": None if view is None else view.get("view_id"),
|
||||
"work_item_type": work_item_type,
|
||||
"included_detail_fields": extra_fields,
|
||||
},
|
||||
"summary": {
|
||||
"success": items_payload.get("success", False),
|
||||
"page": items_payload.get("page"),
|
||||
"total": items_payload.get("total"),
|
||||
},
|
||||
"items": items_payload["data"],
|
||||
}
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--project-key", required=True)
|
||||
parser.add_argument("--user-key", required=True)
|
||||
parser.add_argument("--view-name", required=True)
|
||||
parser.add_argument(
|
||||
"--output",
|
||||
required=True,
|
||||
type=Path,
|
||||
help="Path to the exported JSON file.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--extra-field",
|
||||
action="append",
|
||||
dest="extra_fields",
|
||||
help="Extra detail field to include. Can be repeated.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--work-item-type",
|
||||
default="issue",
|
||||
help="Feishu work item type. Defaults to issue.",
|
||||
)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def main() -> int:
|
||||
args = parse_args()
|
||||
extra_fields = args.extra_fields or list(DEFAULT_EXTRA_FIELDS)
|
||||
|
||||
view = fetch_view(args.project_key, args.user_key, args.view_name, args.work_item_type)
|
||||
items_payload = fetch_items(args.project_key, args.user_key, args.view_name, args.work_item_type)
|
||||
items_payload["data"] = enrich_items(
|
||||
items_payload.get("data", []),
|
||||
args.project_key,
|
||||
args.user_key,
|
||||
args.work_item_type,
|
||||
extra_fields,
|
||||
)
|
||||
|
||||
output = build_output(
|
||||
args.project_key,
|
||||
args.user_key,
|
||||
args.view_name,
|
||||
view,
|
||||
items_payload,
|
||||
extra_fields,
|
||||
args.work_item_type,
|
||||
)
|
||||
|
||||
args.output.parent.mkdir(parents=True, exist_ok=True)
|
||||
args.output.write_text(
|
||||
json.dumps(output, ensure_ascii=False, indent=2) + "\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
66
tools/feishu_project/feishu_project.md
Executable file
66
tools/feishu_project/feishu_project.md
Executable file
@@ -0,0 +1,66 @@
|
||||
`tools/feishu_project` 用来维护飞书问题视图导出、问题数据下载、批量推理,以及问题 case 的后处理脚本。
|
||||
|
||||
当前默认项目配置:
|
||||
- `project_key`: `68ef617fb371dc80a10641f7`
|
||||
- `user_key`: `7550145433285312514`
|
||||
- `view_name`: `董颖-G1Q3`
|
||||
|
||||
推荐入口
|
||||
- 全链路增量同步:`tools/feishu_project/sync_g1q3_issue_data.sh`
|
||||
- CNCAP case 文本清单批处理:`tools/feishu_project/run_cncap_case_pipeline.sh`
|
||||
- 只导出视图:`tools/feishu_project/export_feishu_view_issues.py`
|
||||
- CNCAP case 清单转 synthetic issue JSON:`tools/feishu_project/build_cncap_case_issue_json.py`
|
||||
- 只下载数据:`tools/feishu_project/download_issue_data.sh`
|
||||
- 只跑推理:`tools/feishu_project/run_issue_data_inference.sh`
|
||||
- 问题标签画像统计:`tools/feishu_project/run_issue_tag_profile.sh`
|
||||
- 发布问题标签画像内网页面:`tools/feishu_project/publish_issue_tag_profile_site.sh`
|
||||
- 启动轻量静态服务:`tools/feishu_project/serve_issue_tag_profile_site.sh`
|
||||
- 跟踪与后处理:`tools/feishu_project/run_issue_data_tracking.sh`
|
||||
|
||||
目录约定
|
||||
- 此目录优先放源码和轻量文档,不再推荐把运行产物直接落在这里。
|
||||
- 默认运行产物应写到 `/data1/dongying/Mono3d/*/feishu_project/` 下的 `exports/`、`downloaded_issue_data/`、`inference_issue_data/`、`reports/` 等目录。
|
||||
- `__pycache__/`、HTML 报告、一次性导出 JSON/Markdown 结果属于运行产物或分析产物,除非明确要做样例留存,否则不要继续堆在 `tools/feishu_project/` 根目录。
|
||||
|
||||
维护原则
|
||||
- 通用脚本不要保留单个 case、单个 issue 的硬编码默认值。
|
||||
- G1Q3 包装脚本优先读取同步目录下的最新导出结果,不依赖仓库内静态 JSON。
|
||||
- 复杂逻辑尽量收敛在 Python 主脚本里,Shell 脚本只保留参数拼装和流程编排。
|
||||
|
||||
常用命令
|
||||
```bash
|
||||
# 查看 fp 帮助
|
||||
fp --help
|
||||
|
||||
# 更新 fp
|
||||
fp selfupdate
|
||||
|
||||
# 增量同步 G1Q3 问题并执行下载/推理/跟踪
|
||||
bash tools/feishu_project/sync_g1q3_issue_data.sh
|
||||
|
||||
# 只下载指定 issue
|
||||
bash tools/feishu_project/download_issue_data.sh --issue-id 6965833173
|
||||
|
||||
# 只对指定 issue 跑推理
|
||||
bash tools/feishu_project/run_issue_data_inference.sh --issue-id 6965833173
|
||||
|
||||
# 基于最新 G1Q3 导出生成目标/问题标签画像
|
||||
bash tools/feishu_project/run_issue_tag_profile.sh
|
||||
|
||||
# 生成标签画像并发布到静态站点目录
|
||||
bash tools/feishu_project/publish_issue_tag_profile_site.sh
|
||||
|
||||
# 无 Nginx 时临时启动内网静态服务
|
||||
bash tools/feishu_project/serve_issue_tag_profile_site.sh start
|
||||
|
||||
# 指定内网访问地址。该 IP 必须是当前机器网卡 IP,或由反向代理/内网服务器承载。
|
||||
INTRANET_HOST=192.168.2.169 bash tools/feishu_project/publish_issue_tag_profile_site.sh
|
||||
INTRANET_HOST=192.168.2.169 bash tools/feishu_project/serve_issue_tag_profile_site.sh restart
|
||||
```
|
||||
|
||||
内网页面发布
|
||||
- 默认发布路径:`/data1/dongying/Mono3d/G1Q3/feishu_project/site/issue_tag_profile/index.html`
|
||||
- 默认访问路径:`http://<内网机器IP>:8088/issue_tag_profile/`
|
||||
- Nginx 示例配置:`tools/feishu_project/nginx_issue_tag_profile.conf.example`
|
||||
- 临时静态服务脚本:`tools/feishu_project/serve_issue_tag_profile_site.sh`
|
||||
- 可用 `INTRANET_HOST=<内网IP>` 或 `PUBLIC_HOST=<内网IP>` 指定页面里展示的访问地址;如果该 IP 不在当前机器网卡上,脚本会提示需要部署到对应内网机器或配置反向代理。
|
||||
178
tools/feishu_project/recover_downloaded_issue_calibs.py
Executable file
178
tools/feishu_project/recover_downloaded_issue_calibs.py
Executable file
@@ -0,0 +1,178 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Recover missing camera4.json files for downloaded standard-path issue data."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import sys
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
FILE = Path(__file__).resolve()
|
||||
ROOT = FILE.parents[2]
|
||||
if str(ROOT) not in sys.path:
|
||||
sys.path.append(str(ROOT))
|
||||
|
||||
from tools.feishu_project.case_calib_recovery import CalibRecoveryResult, recover_camera4_json
|
||||
|
||||
|
||||
DEFAULT_DOWNLOAD_ROOT = Path("/data1/dongying/Mono3d/G1Q3/feishu_project/downloaded_issue_data")
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument(
|
||||
"--download-root",
|
||||
default=str(DEFAULT_DOWNLOAD_ROOT),
|
||||
help="Root directory containing downloaded issue data.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--issue-id",
|
||||
action="append",
|
||||
dest="issue_ids",
|
||||
type=int,
|
||||
help="Only process the specified issue id. Can be repeated.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--path-root",
|
||||
action="append",
|
||||
default=[],
|
||||
help=(
|
||||
"Optional explicit copied target root to recover. Can be repeated. "
|
||||
"Example: downloaded_issue_data/issue_x/path_01/case_dir"
|
||||
),
|
||||
)
|
||||
parser.add_argument(
|
||||
"--manifest-path",
|
||||
default="",
|
||||
help="Optional explicit manifest path. Defaults to <download-root>/calib_recovery_manifest.json",
|
||||
)
|
||||
parser.add_argument("--dry-run", action="store_true")
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def log_progress(message: str) -> None:
|
||||
timestamp = datetime.now().astimezone().strftime("%Y-%m-%d %H:%M:%S")
|
||||
print(f"[recover_downloaded_issue_calibs {timestamp}] {message}", flush=True)
|
||||
|
||||
|
||||
def discover_target_roots(download_root: Path, issue_ids: list[int], explicit_roots: list[str]) -> list[Path]:
|
||||
if explicit_roots:
|
||||
return sorted({Path(path).resolve() for path in explicit_roots})
|
||||
|
||||
issue_dirs: list[Path]
|
||||
if issue_ids:
|
||||
issue_dirs = [download_root / f"issue_{issue_id}" for issue_id in sorted(set(issue_ids))]
|
||||
else:
|
||||
issue_dirs = sorted(path for path in download_root.glob("issue_*") if path.is_dir())
|
||||
|
||||
targets: list[Path] = []
|
||||
seen: set[Path] = set()
|
||||
for issue_dir in issue_dirs:
|
||||
if not issue_dir.is_dir():
|
||||
continue
|
||||
for path_dir in sorted(path for path in issue_dir.glob("path_*") if path.is_dir()):
|
||||
direct_case = path_dir / "sigmastar.1" / "camera4.bin"
|
||||
if direct_case.is_file() and path_dir not in seen:
|
||||
seen.add(path_dir)
|
||||
targets.append(path_dir)
|
||||
|
||||
for child in sorted(path_dir.iterdir()):
|
||||
if not child.is_dir() or child.name == "test_data":
|
||||
continue
|
||||
if child in seen:
|
||||
continue
|
||||
seen.add(child)
|
||||
targets.append(child)
|
||||
return targets
|
||||
|
||||
|
||||
def result_to_dict(target_root: Path, result: CalibRecoveryResult) -> dict:
|
||||
return {
|
||||
"target_root": str(target_root),
|
||||
"target_path": str(result.target_path),
|
||||
"status": result.status,
|
||||
"detail": result.detail,
|
||||
"source_path": None if result.source_path is None else str(result.source_path),
|
||||
"source_kind": result.source_kind,
|
||||
}
|
||||
|
||||
|
||||
def build_manifest(
|
||||
download_root: Path,
|
||||
dry_run: bool,
|
||||
issue_ids: list[int],
|
||||
target_roots: list[Path],
|
||||
results: list[dict],
|
||||
) -> dict:
|
||||
summary: dict[str, int] = {}
|
||||
for result in results:
|
||||
status = result["status"]
|
||||
summary[status] = summary.get(status, 0) + 1
|
||||
|
||||
return {
|
||||
"generated_at": datetime.now().astimezone().isoformat(timespec="seconds"),
|
||||
"download_root": str(download_root),
|
||||
"dry_run": dry_run,
|
||||
"issue_filter": issue_ids,
|
||||
"target_roots": [str(path) for path in target_roots],
|
||||
"summary": summary,
|
||||
"results": results,
|
||||
}
|
||||
|
||||
|
||||
def print_summary(manifest: dict) -> None:
|
||||
print(f"download_root: {manifest['download_root']}")
|
||||
print(f"dry_run: {manifest['dry_run']}")
|
||||
print(f"target_roots: {len(manifest['target_roots'])}")
|
||||
for status, count in sorted(manifest["summary"].items()):
|
||||
print(f"{status}: {count}")
|
||||
|
||||
|
||||
def main() -> int:
|
||||
args = parse_args()
|
||||
download_root = Path(args.download_root).resolve()
|
||||
manifest_path = (
|
||||
Path(args.manifest_path).resolve()
|
||||
if args.manifest_path
|
||||
else download_root / "calib_recovery_manifest.json"
|
||||
)
|
||||
target_roots = discover_target_roots(download_root, args.issue_ids or [], args.path_root)
|
||||
log_progress(f"targets_to_process: {len(target_roots)}")
|
||||
|
||||
results: list[dict] = []
|
||||
for index, target_root in enumerate(target_roots, start=1):
|
||||
log_progress(f"[{index}/{len(target_roots)}] start: {target_root}")
|
||||
recovery = recover_camera4_json(
|
||||
source_root=target_root,
|
||||
target_root=target_root,
|
||||
dry_run=args.dry_run,
|
||||
)
|
||||
results.append(result_to_dict(target_root, recovery))
|
||||
log_progress(f"[{index}/{len(target_roots)}] done: {recovery.status}")
|
||||
|
||||
manifest = build_manifest(
|
||||
download_root=download_root,
|
||||
dry_run=args.dry_run,
|
||||
issue_ids=args.issue_ids or [],
|
||||
target_roots=target_roots,
|
||||
results=results,
|
||||
)
|
||||
if not args.dry_run:
|
||||
manifest_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
manifest_path.write_text(
|
||||
json.dumps(manifest, ensure_ascii=False, indent=2) + "\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
print_summary(manifest)
|
||||
if args.dry_run:
|
||||
print(f"manifest (not written in dry-run): {manifest_path}")
|
||||
else:
|
||||
print(f"manifest: {manifest_path}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
330
tools/feishu_project/run_cncap_case_pipeline.sh
Executable file
330
tools/feishu_project/run_cncap_case_pipeline.sh
Executable file
@@ -0,0 +1,330 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
|
||||
PROJECT_ROOT=$(cd "${SCRIPT_DIR}/../.." && pwd)
|
||||
|
||||
CASE_LIST=${CASE_LIST:-"${PROJECT_ROOT}/tools/feishu_project/cncap_case.txt"}
|
||||
SYNC_ROOT=${SYNC_ROOT:-/data1/dongying/Mono3d/CNCAP/feishu_project}
|
||||
EXPORT_JSON=${EXPORT_JSON:-"${SYNC_ROOT}/exports/cncap_case_issue_list.json"}
|
||||
CASE_INDEX_JSON=${CASE_INDEX_JSON:-"${SYNC_ROOT}/exports/cncap_case_issue_index.json"}
|
||||
DOWNLOAD_ROOT=${DOWNLOAD_ROOT:-"${SYNC_ROOT}/downloaded_issue_data"}
|
||||
DOWNLOAD_MANIFEST_PATH=${DOWNLOAD_MANIFEST_PATH:-"${DOWNLOAD_ROOT}/download_manifest.json"}
|
||||
INFERENCE_ROOT=${INFERENCE_ROOT:-"${SYNC_ROOT}/inference_issue_data"}
|
||||
INFERENCE_MANIFEST_PATH=${INFERENCE_MANIFEST_PATH:-"${INFERENCE_ROOT}/inference_manifest.json"}
|
||||
|
||||
PYTHON_BIN=${PYTHON_BIN:-/deeplearning_team/ydong/dongying/miniconda/envs/dev/bin/python}
|
||||
RUN_BUILD=${RUN_BUILD:-1}
|
||||
RUN_DOWNLOAD=${RUN_DOWNLOAD:-1}
|
||||
RUN_INFERENCE=${RUN_INFERENCE:-1}
|
||||
ENABLE_TRACKING=${ENABLE_TRACKING:-1}
|
||||
ENABLE_CONVERT=${ENABLE_CONVERT:-1}
|
||||
ENABLE_VISUALIZE=${ENABLE_VISUALIZE:-0}
|
||||
ENABLE_TEMPORAL_ANALYSIS=${ENABLE_TEMPORAL_ANALYSIS:-0}
|
||||
SKIP_EXISTING_INFERENCE=${SKIP_EXISTING_INFERENCE:-1}
|
||||
DRY_RUN=${DRY_RUN:-0}
|
||||
|
||||
USE_ISSUE_FRAME_WINDOW=${USE_ISSUE_FRAME_WINDOW:-0}
|
||||
FRAME_BEFORE=${FRAME_BEFORE:-200}
|
||||
FRAME_AFTER=${FRAME_AFTER:-200}
|
||||
MISSING_ISSUE_FRAME_POLICY=${MISSING_ISSUE_FRAME_POLICY:-full}
|
||||
FRAME_INDEX_START=${FRAME_INDEX_START:-}
|
||||
FRAME_INDEX_END=${FRAME_INDEX_END:-}
|
||||
FRAME_ID_START=${FRAME_ID_START:-}
|
||||
FRAME_ID_END=${FRAME_ID_END:-}
|
||||
TARGET_FRAME_ID=${TARGET_FRAME_ID:-}
|
||||
|
||||
VIDEO_STRIDE=${VIDEO_STRIDE:-1}
|
||||
MAX_IMAGES=${MAX_IMAGES:-0}
|
||||
EXPORTED_MODEL=${EXPORTED_MODEL:-${PROJECT_ROOT}/runs/export/train_mono3d_two_roi_20260416-raw_no_edge/merged_model.torchscript}
|
||||
DEVICE=${DEVICE:-}
|
||||
PROVIDERS=${PROVIDERS:-}
|
||||
ENABLE_ATTR=${ENABLE_ATTR:-0}
|
||||
ENABLE_CROSS_CLASS_MERGE_PRIOR=${ENABLE_CROSS_CLASS_MERGE_PRIOR:-1}
|
||||
SAVE_AGGREGATE_PREDICTIONS=${SAVE_AGGREGATE_PREDICTIONS:-1}
|
||||
INFERENCE_EXTRA_ARGS=${INFERENCE_EXTRA_ARGS:-}
|
||||
|
||||
TRACK_CLASSES=${TRACK_CLASSES:-0 1 2 3 4 5 6 7 8 9 10 11 12}
|
||||
IOU_THRESH=${IOU_THRESH:-0.3}
|
||||
MAX_AGE=${MAX_AGE:-5}
|
||||
MIN_HITS=${MIN_HITS:-1}
|
||||
DIST_THRESH=${DIST_THRESH:-100}
|
||||
MAX_3D_DISTANCE=${MAX_3D_DISTANCE:-10.0}
|
||||
MAX_FRAMES=${MAX_FRAMES:-}
|
||||
MERGE_OUTPUT_NAME=${MERGE_OUTPUT_NAME:-combined_tracking.json}
|
||||
FILE_PATTERN=${FILE_PATTERN:-*.json}
|
||||
ENABLE_USE_3D=${ENABLE_USE_3D:-0}
|
||||
CONVERT_OUTPUT_DIR_NAME=${CONVERT_OUTPUT_DIR_NAME:-objectlist}
|
||||
CONVERT_TRACKING_JSON_NAME=${CONVERT_TRACKING_JSON_NAME:-merge.json}
|
||||
CONVERT_CAM_ID=${CONVERT_CAM_ID:-}
|
||||
VIS_DOWNLOAD_ROOT=${VIS_DOWNLOAD_ROOT:-"${DOWNLOAD_ROOT}"}
|
||||
VIS_OUTPUT_ROOT=${VIS_OUTPUT_ROOT:-}
|
||||
VIS_OUTPUT_DIR_NAME=${VIS_OUTPUT_DIR_NAME:-tracking_vis_raw}
|
||||
VIS_TRACKING_JSON_NAME=${VIS_TRACKING_JSON_NAME:-merge.json}
|
||||
VIS_MAX_FRAMES=${VIS_MAX_FRAMES:-}
|
||||
VIS_CLASS_ID=${VIS_CLASS_ID:-}
|
||||
VIS_TRACK_IDS=${VIS_TRACK_IDS:-}
|
||||
VIS_SHOW_TRAJECTORY=${VIS_SHOW_TRAJECTORY:-0}
|
||||
TEMPORAL_OUTPUT_ROOT=${TEMPORAL_OUTPUT_ROOT:-}
|
||||
TEMPORAL_OUTPUT_DIR_NAME=${TEMPORAL_OUTPUT_DIR_NAME:-temporal_observation}
|
||||
TEMPORAL_JSON_NAME=${TEMPORAL_JSON_NAME:-merge.json}
|
||||
TEMPORAL_MIN_LENGTH=${TEMPORAL_MIN_LENGTH:-3}
|
||||
TEMPORAL_TRACK_IDS=${TEMPORAL_TRACK_IDS:-}
|
||||
TEMPORAL_CLASS_ID=${TEMPORAL_CLASS_ID:-}
|
||||
TEMPORAL_FRAME_ID_START=${TEMPORAL_FRAME_ID_START:-}
|
||||
TEMPORAL_FRAME_ID_END=${TEMPORAL_FRAME_ID_END:-}
|
||||
TEMPORAL_PLOTS=${TEMPORAL_PLOTS:-0}
|
||||
TEMPORAL_EXPORT_SERIES=${TEMPORAL_EXPORT_SERIES:-1}
|
||||
TEMPORAL_FOCUS_TRACK_PLOTS=${TEMPORAL_FOCUS_TRACK_PLOTS:-1}
|
||||
TEMPORAL_PREFER_EGO=${TEMPORAL_PREFER_EGO:-1}
|
||||
TEMPORAL_X_AXIS=${TEMPORAL_X_AXIS:-frame_id}
|
||||
TEMPORAL_HEADING_SOURCE=${TEMPORAL_HEADING_SOURCE:-camera_reg}
|
||||
TRACKING_MODEL_VERSION=${TRACKING_MODEL_VERSION:-}
|
||||
|
||||
SKIP_MDI=${SKIP_MDI:-0}
|
||||
SKIP_COPY=${SKIP_COPY:-0}
|
||||
ONLY_REDOWNLOAD_AFFECTED_CASES=${ONLY_REDOWNLOAD_AFFECTED_CASES:-0}
|
||||
ID_BASE=${ID_BASE:-9000000000}
|
||||
|
||||
CASE_JSON_BUILDER=${CASE_JSON_BUILDER:-"${PROJECT_ROOT}/tools/feishu_project/build_cncap_case_issue_json.py"}
|
||||
DOWNLOAD_WRAPPER=${DOWNLOAD_WRAPPER:-"${PROJECT_ROOT}/tools/feishu_project/download_issue_data.sh"}
|
||||
INFERENCE_WRAPPER=${INFERENCE_WRAPPER:-"${PROJECT_ROOT}/tools/feishu_project/run_issue_data_inference.sh"}
|
||||
TRACKING_WRAPPER=${TRACKING_WRAPPER:-"${PROJECT_ROOT}/tools/feishu_project/run_issue_data_tracking.sh"}
|
||||
|
||||
ISSUE_ID_ARGS=()
|
||||
ISSUE_ID_VALUES=()
|
||||
|
||||
show_usage() {
|
||||
cat <<EOF
|
||||
Usage:
|
||||
bash $(basename "${BASH_SOURCE[0]}") [--issue-id <id>]...
|
||||
|
||||
Environment variables:
|
||||
CASE_LIST, SYNC_ROOT, EXPORT_JSON, CASE_INDEX_JSON
|
||||
DOWNLOAD_ROOT, INFERENCE_ROOT, EXPORTED_MODEL
|
||||
RUN_BUILD, RUN_DOWNLOAD, RUN_INFERENCE, ENABLE_TRACKING
|
||||
ENABLE_CONVERT, ENABLE_VISUALIZE, ENABLE_TEMPORAL_ANALYSIS
|
||||
DRY_RUN, ID_BASE
|
||||
EOF
|
||||
}
|
||||
|
||||
while (($# > 0)); do
|
||||
case "$1" in
|
||||
--issue-id)
|
||||
if (($# < 2)); then
|
||||
echo "Error: --issue-id requires a value" >&2
|
||||
exit 1
|
||||
fi
|
||||
ISSUE_ID_ARGS+=("$1" "$2")
|
||||
ISSUE_ID_VALUES+=("$2")
|
||||
shift 2
|
||||
;;
|
||||
--issue-id=*)
|
||||
issue_id="${1#*=}"
|
||||
ISSUE_ID_ARGS+=("$1")
|
||||
ISSUE_ID_VALUES+=("${issue_id}")
|
||||
shift
|
||||
;;
|
||||
-h|--help)
|
||||
show_usage
|
||||
exit 0
|
||||
;;
|
||||
*)
|
||||
echo "Error: unsupported option: $1" >&2
|
||||
show_usage >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
print_stage() {
|
||||
local stage_name="$1"
|
||||
echo ""
|
||||
echo "######################################################################"
|
||||
echo "# ${stage_name}"
|
||||
echo "######################################################################"
|
||||
}
|
||||
|
||||
resolve_tracking_model_version() {
|
||||
if [[ -n "${TRACKING_MODEL_VERSION}" ]]; then
|
||||
printf '%s\n' "${TRACKING_MODEL_VERSION}"
|
||||
return 0
|
||||
fi
|
||||
|
||||
if [[ -n "${EXPORTED_MODEL}" ]] && [[ "${EXPORTED_MODEL}" =~ ([0-9]{8}) ]]; then
|
||||
printf '%s\n' "${BASH_REMATCH[1]}"
|
||||
return 0
|
||||
fi
|
||||
|
||||
return 1
|
||||
}
|
||||
|
||||
has_local_video_cases() {
|
||||
local search_root="$1"
|
||||
shift || true
|
||||
|
||||
if [[ ! -d "${search_root}" ]]; then
|
||||
return 1
|
||||
fi
|
||||
|
||||
if [[ "$#" -eq 0 ]]; then
|
||||
[[ -n "$(find "${search_root}" -type f -path '*/sigmastar.1/camera4.bin' -print -quit)" ]]
|
||||
return
|
||||
fi
|
||||
|
||||
local issue_id=""
|
||||
for issue_id in "$@"; do
|
||||
if [[ -d "${search_root}/issue_${issue_id}" ]] && [[ -n "$(find "${search_root}/issue_${issue_id}" -type f -path '*/sigmastar.1/camera4.bin' -print -quit)" ]]; then
|
||||
return 0
|
||||
fi
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
TMP_ROOT=""
|
||||
EFFECTIVE_EXPORT_JSON="${EXPORT_JSON}"
|
||||
EFFECTIVE_CASE_INDEX_JSON="${CASE_INDEX_JSON}"
|
||||
|
||||
cleanup() {
|
||||
if [[ -n "${TMP_ROOT}" && -d "${TMP_ROOT}" ]]; then
|
||||
rm -rf "${TMP_ROOT}"
|
||||
fi
|
||||
}
|
||||
trap cleanup EXIT
|
||||
|
||||
if [[ "${DRY_RUN}" == "1" && "${RUN_BUILD}" == "1" ]]; then
|
||||
TMP_ROOT=$(mktemp -d /tmp/cncap_case_pipeline.XXXXXX)
|
||||
EFFECTIVE_EXPORT_JSON="${TMP_ROOT}/cncap_case_issue_list.json"
|
||||
EFFECTIVE_CASE_INDEX_JSON="${TMP_ROOT}/cncap_case_issue_index.json"
|
||||
fi
|
||||
|
||||
export PYTHON_BIN
|
||||
export TRACK_CLASSES
|
||||
export IOU_THRESH
|
||||
export MAX_AGE
|
||||
export MIN_HITS
|
||||
export DIST_THRESH
|
||||
export MAX_3D_DISTANCE
|
||||
export MAX_FRAMES
|
||||
export MERGE_OUTPUT_NAME
|
||||
export FILE_PATTERN
|
||||
export ENABLE_USE_3D
|
||||
export ENABLE_CONVERT
|
||||
export ENABLE_VISUALIZE
|
||||
export ENABLE_TEMPORAL_ANALYSIS
|
||||
export CONVERT_OUTPUT_DIR_NAME
|
||||
export CONVERT_TRACKING_JSON_NAME
|
||||
export CONVERT_CAM_ID
|
||||
export VIS_DOWNLOAD_ROOT
|
||||
export VIS_OUTPUT_ROOT
|
||||
export VIS_OUTPUT_DIR_NAME
|
||||
export VIS_TRACKING_JSON_NAME
|
||||
export VIS_MAX_FRAMES
|
||||
export VIS_CLASS_ID
|
||||
export VIS_TRACK_IDS
|
||||
export VIS_SHOW_TRAJECTORY
|
||||
export TEMPORAL_OUTPUT_ROOT
|
||||
export TEMPORAL_OUTPUT_DIR_NAME
|
||||
export TEMPORAL_JSON_NAME
|
||||
export TEMPORAL_MIN_LENGTH
|
||||
export TEMPORAL_TRACK_IDS
|
||||
export TEMPORAL_CLASS_ID
|
||||
export TEMPORAL_FRAME_ID_START
|
||||
export TEMPORAL_FRAME_ID_END
|
||||
export TEMPORAL_PLOTS
|
||||
export TEMPORAL_EXPORT_SERIES
|
||||
export TEMPORAL_FOCUS_TRACK_PLOTS
|
||||
export TEMPORAL_PREFER_EGO
|
||||
export TEMPORAL_X_AXIS
|
||||
export TEMPORAL_HEADING_SOURCE
|
||||
|
||||
if [[ "${RUN_BUILD}" == "1" ]]; then
|
||||
print_stage "Build CNCAP Synthetic Issue JSON"
|
||||
echo "CASE_LIST : ${CASE_LIST}"
|
||||
echo "EXPORT_JSON : ${EFFECTIVE_EXPORT_JSON}"
|
||||
echo "CASE_INDEX_JSON : ${EFFECTIVE_CASE_INDEX_JSON}"
|
||||
echo "ID_BASE : ${ID_BASE}"
|
||||
|
||||
"${PYTHON_BIN}" "${CASE_JSON_BUILDER}" \
|
||||
--input "${CASE_LIST}" \
|
||||
--output "${EFFECTIVE_EXPORT_JSON}" \
|
||||
--case-index-output "${EFFECTIVE_CASE_INDEX_JSON}" \
|
||||
--id-base "${ID_BASE}"
|
||||
elif [[ ! -f "${EFFECTIVE_EXPORT_JSON}" ]]; then
|
||||
echo "Error: export json does not exist and RUN_BUILD=0: ${EFFECTIVE_EXPORT_JSON}" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [[ "${RUN_DOWNLOAD}" == "1" ]]; then
|
||||
print_stage "Download CNCAP Case Data"
|
||||
INPUT_JSON="${EFFECTIVE_EXPORT_JSON}" \
|
||||
OUTPUT_ROOT="${DOWNLOAD_ROOT}" \
|
||||
MANIFEST_PATH="${DOWNLOAD_MANIFEST_PATH}" \
|
||||
PYTHON_BIN="${PYTHON_BIN}" \
|
||||
DRY_RUN="${DRY_RUN}" \
|
||||
SKIP_MDI="${SKIP_MDI}" \
|
||||
SKIP_COPY="${SKIP_COPY}" \
|
||||
ONLY_REDOWNLOAD_AFFECTED_CASES="${ONLY_REDOWNLOAD_AFFECTED_CASES}" \
|
||||
bash "${DOWNLOAD_WRAPPER}" "${ISSUE_ID_ARGS[@]}"
|
||||
fi
|
||||
|
||||
if [[ "${RUN_INFERENCE}" == "1" ]]; then
|
||||
if [[ "${DRY_RUN}" == "1" ]] && ! has_local_video_cases "${DOWNLOAD_ROOT}" "${ISSUE_ID_VALUES[@]}"; then
|
||||
print_stage "Skip CNCAP Inference"
|
||||
echo "Skipping inference in dry-run because no local downloaded camera4.bin cases were found."
|
||||
echo "Run download without DRY_RUN first, or reuse an existing DOWNLOAD_ROOT to plan inference."
|
||||
else
|
||||
print_stage "Run CNCAP Inference"
|
||||
DOWNLOAD_ROOT="${DOWNLOAD_ROOT}" \
|
||||
OUTPUT_ROOT="${INFERENCE_ROOT}" \
|
||||
MANIFEST_PATH="${INFERENCE_MANIFEST_PATH}" \
|
||||
PYTHON_BIN="${PYTHON_BIN}" \
|
||||
ISSUE_JSON="${EFFECTIVE_EXPORT_JSON}" \
|
||||
VIDEO_STRIDE="${VIDEO_STRIDE}" \
|
||||
MAX_IMAGES="${MAX_IMAGES}" \
|
||||
FRAME_INDEX_START="${FRAME_INDEX_START}" \
|
||||
FRAME_INDEX_END="${FRAME_INDEX_END}" \
|
||||
FRAME_ID_START="${FRAME_ID_START}" \
|
||||
FRAME_ID_END="${FRAME_ID_END}" \
|
||||
TARGET_FRAME_ID="${TARGET_FRAME_ID}" \
|
||||
FRAME_BEFORE="${FRAME_BEFORE}" \
|
||||
FRAME_AFTER="${FRAME_AFTER}" \
|
||||
USE_ISSUE_FRAME_WINDOW="${USE_ISSUE_FRAME_WINDOW}" \
|
||||
MISSING_ISSUE_FRAME_POLICY="${MISSING_ISSUE_FRAME_POLICY}" \
|
||||
EXPORTED_MODEL="${EXPORTED_MODEL}" \
|
||||
DEVICE="${DEVICE}" \
|
||||
PROVIDERS="${PROVIDERS}" \
|
||||
ENABLE_ATTR="${ENABLE_ATTR}" \
|
||||
ENABLE_CROSS_CLASS_MERGE_PRIOR="${ENABLE_CROSS_CLASS_MERGE_PRIOR}" \
|
||||
SAVE_AGGREGATE_PREDICTIONS="${SAVE_AGGREGATE_PREDICTIONS}" \
|
||||
SKIP_EXISTING="${SKIP_EXISTING_INFERENCE}" \
|
||||
DRY_RUN="${DRY_RUN}" \
|
||||
ENABLE_TRACKING=0 \
|
||||
INFERENCE_EXTRA_ARGS="${INFERENCE_EXTRA_ARGS}" \
|
||||
bash "${INFERENCE_WRAPPER}" "${ISSUE_ID_ARGS[@]}"
|
||||
fi
|
||||
fi
|
||||
|
||||
if [[ "${ENABLE_TRACKING}" != "1" ]]; then
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if [[ "${DRY_RUN}" == "1" ]]; then
|
||||
print_stage "Skip CNCAP Tracking"
|
||||
echo "Skipping tracking because DRY_RUN=1."
|
||||
exit 0
|
||||
fi
|
||||
|
||||
TRACKING_MODEL_VERSION_RESOLVED=$(resolve_tracking_model_version || true)
|
||||
|
||||
print_stage "Run CNCAP Tracking/Convert Workflow"
|
||||
echo "INFERENCE_ROOT : ${INFERENCE_ROOT}"
|
||||
echo "VIS_DOWNLOAD_ROOT : ${VIS_DOWNLOAD_ROOT}"
|
||||
if [[ -n "${TRACKING_MODEL_VERSION_RESOLVED}" ]]; then
|
||||
echo "TRACKING_MODEL_VERSION : ${TRACKING_MODEL_VERSION_RESOLVED}"
|
||||
fi
|
||||
|
||||
RESULTS_ROOT="${INFERENCE_ROOT}" \
|
||||
PYTHON_BIN="${PYTHON_BIN}" \
|
||||
TRACKING_MODEL_VERSION="${TRACKING_MODEL_VERSION_RESOLVED}" \
|
||||
bash "${TRACKING_WRAPPER}" "${INFERENCE_ROOT}" "${ISSUE_ID_ARGS[@]}"
|
||||
186
tools/feishu_project/run_issue_data_convert_tracking.sh
Executable file
186
tools/feishu_project/run_issue_data_convert_tracking.sh
Executable file
@@ -0,0 +1,186 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
|
||||
PROJECT_ROOT=$(cd "${SCRIPT_DIR}/../.." && pwd)
|
||||
|
||||
RESULTS_ROOT=${RESULTS_ROOT:-/data1/dongying/Mono3d/G1Q3/feishu_project/inference_issue_data}
|
||||
OUTPUT_DIR_NAME=${OUTPUT_DIR_NAME:-objectlist}
|
||||
PYTHON_BIN=${PYTHON_BIN:-/deeplearning_team/ydong/dongying/miniconda/envs/dev/bin/python}
|
||||
CONVERT_LAUNCHER=${CONVERT_LAUNCHER:-"${PROJECT_ROOT}/tools/convert_merge_tracking_bundle/convert_merge_tracking_exported_onnx_infer_case.sh"}
|
||||
MERGE_JSON_NAME=${MERGE_JSON_NAME:-merge.json}
|
||||
CAM_ID=${CAM_ID:-}
|
||||
|
||||
TARGET_PATH=""
|
||||
ISSUE_IDS=()
|
||||
|
||||
while (($# > 0)); do
|
||||
case "$1" in
|
||||
--issue-id)
|
||||
if (($# < 2)); then
|
||||
echo "Error: --issue-id requires a value" >&2
|
||||
exit 1
|
||||
fi
|
||||
ISSUE_IDS+=("$2")
|
||||
shift 2
|
||||
;;
|
||||
--issue-id=*)
|
||||
ISSUE_IDS+=("${1#*=}")
|
||||
shift
|
||||
;;
|
||||
-*)
|
||||
echo "Error: unsupported option: $1" >&2
|
||||
exit 1
|
||||
;;
|
||||
*)
|
||||
if [[ -n "${TARGET_PATH}" ]]; then
|
||||
echo "Error: multiple target paths provided: ${TARGET_PATH} and $1" >&2
|
||||
exit 1
|
||||
fi
|
||||
TARGET_PATH="$1"
|
||||
shift
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
if [[ -z "${TARGET_PATH}" ]]; then
|
||||
TARGET_PATH="${RESULTS_ROOT}"
|
||||
fi
|
||||
|
||||
if [[ ! -e "${TARGET_PATH}" ]]; then
|
||||
echo "Error: target path does not exist: ${TARGET_PATH}" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
is_case_dir() {
|
||||
local dir_path="$1"
|
||||
[[ -d "${dir_path}" ]] && [[ -f "${dir_path}/${MERGE_JSON_NAME}" ]]
|
||||
}
|
||||
|
||||
is_predictions_dir() {
|
||||
local dir_path="$1"
|
||||
[[ -d "${dir_path}" ]] || return 1
|
||||
is_case_dir "$(dirname "${dir_path}")"
|
||||
}
|
||||
|
||||
resolve_case_dir() {
|
||||
local target_path="$1"
|
||||
|
||||
if is_case_dir "${target_path}"; then
|
||||
printf '%s\n' "${target_path}"
|
||||
return 0
|
||||
fi
|
||||
|
||||
if is_predictions_dir "${target_path}"; then
|
||||
dirname "${target_path}"
|
||||
return 0
|
||||
fi
|
||||
|
||||
if [[ -f "${target_path}" ]] && [[ "$(basename "${target_path}")" == "${MERGE_JSON_NAME}" ]]; then
|
||||
dirname "${target_path}"
|
||||
return 0
|
||||
fi
|
||||
|
||||
return 1
|
||||
}
|
||||
|
||||
case_output_dir() {
|
||||
local case_dir="$1"
|
||||
printf '%s/%s\n' "${case_dir%/}" "${OUTPUT_DIR_NAME}"
|
||||
}
|
||||
|
||||
run_convert_case() {
|
||||
local case_dir="$1"
|
||||
local output_path
|
||||
|
||||
output_path="$(case_output_dir "${case_dir}")"
|
||||
|
||||
echo ""
|
||||
echo "######################################################################"
|
||||
echo "# Feishu issue-data protocol conversion"
|
||||
echo "######################################################################"
|
||||
echo "Case dir : ${case_dir}"
|
||||
echo "Tracking JSON: ${case_dir%/}/${MERGE_JSON_NAME}"
|
||||
echo "Output path : ${output_path}"
|
||||
if [[ -n "${CAM_ID}" ]]; then
|
||||
echo "Camera ID override: ${CAM_ID}"
|
||||
fi
|
||||
|
||||
PYTHON_BIN="${PYTHON_BIN}" \
|
||||
MERGE_JSON_NAME="${MERGE_JSON_NAME}" \
|
||||
CAM_ID="${CAM_ID}" \
|
||||
bash "${CONVERT_LAUNCHER}" "${case_dir}" "${output_path}"
|
||||
}
|
||||
|
||||
run_convert_root() {
|
||||
local target_root="$1"
|
||||
local total_cases=0
|
||||
local success_cases=0
|
||||
local failed_cases=0
|
||||
|
||||
echo ""
|
||||
echo "Batch-root mode: ${target_root}"
|
||||
|
||||
while IFS= read -r -d '' merge_json_path; do
|
||||
local case_dir
|
||||
case_dir="$(dirname "${merge_json_path}")"
|
||||
((total_cases += 1))
|
||||
|
||||
if run_convert_case "${case_dir}"; then
|
||||
((success_cases += 1))
|
||||
else
|
||||
((failed_cases += 1))
|
||||
printf '[FAIL] case=%s\n' "${case_dir}" >&2
|
||||
fi
|
||||
done < <(
|
||||
find "${target_root}" -type f -name "${MERGE_JSON_NAME}" -print0 | sort -z
|
||||
)
|
||||
|
||||
if [[ "${total_cases}" -eq 0 ]]; then
|
||||
echo "[ERROR] No ${MERGE_JSON_NAME} files were found under: ${target_root}" >&2
|
||||
return 1
|
||||
fi
|
||||
|
||||
echo ""
|
||||
printf '[DONE] cases=%d success=%d failed=%d\n' \
|
||||
"${total_cases}" "${success_cases}" "${failed_cases}"
|
||||
|
||||
[[ "${failed_cases}" -eq 0 ]]
|
||||
}
|
||||
|
||||
if [[ "${#ISSUE_IDS[@]}" -eq 0 ]]; then
|
||||
if RESOLVED_CASE_DIR="$(resolve_case_dir "${TARGET_PATH}")"; then
|
||||
run_convert_case "${RESOLVED_CASE_DIR}"
|
||||
else
|
||||
run_convert_root "${TARGET_PATH}"
|
||||
fi
|
||||
exit 0
|
||||
fi
|
||||
|
||||
total_issues=0
|
||||
success_issues=0
|
||||
failed_issues=0
|
||||
|
||||
for issue_id in "${ISSUE_IDS[@]}"; do
|
||||
issue_root="${TARGET_PATH}/issue_${issue_id}"
|
||||
((total_issues += 1))
|
||||
|
||||
if [[ ! -d "${issue_root}" ]]; then
|
||||
echo "[FAIL] issue_${issue_id}: not found under ${TARGET_PATH}" >&2
|
||||
((failed_issues += 1))
|
||||
continue
|
||||
fi
|
||||
|
||||
if run_convert_root "${issue_root}"; then
|
||||
((success_issues += 1))
|
||||
else
|
||||
((failed_issues += 1))
|
||||
echo "[FAIL] issue_${issue_id}: protocol conversion failed" >&2
|
||||
fi
|
||||
done
|
||||
|
||||
echo ""
|
||||
printf '[DONE] issues=%d success=%d failed=%d\n' \
|
||||
"${total_issues}" "${success_issues}" "${failed_issues}"
|
||||
|
||||
[[ "${failed_issues}" -eq 0 ]]
|
||||
721
tools/feishu_project/run_issue_data_inference.py
Executable file
721
tools/feishu_project/run_issue_data_inference.py
Executable file
@@ -0,0 +1,721 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Batch inference runner for downloaded Feishu issue data."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Iterable
|
||||
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
DEFAULT_DOWNLOAD_ROOT = Path("/data1/dongying/Mono3d/G1Q3/feishu_project/downloaded_issue_data")
|
||||
DEFAULT_OUTPUT_ROOT = Path("/data1/dongying/Mono3d/G1Q3/feishu_project/inference_issue_data")
|
||||
DEFAULT_INFERENCE_SCRIPT = ROOT / "tools" / "model_inference" / "core" / "run_two_roi_exported_onnx_infer.py"
|
||||
DEFAULT_PYTHON_BIN = Path("/deeplearning_team/ydong/dongying/miniconda/envs/dev/bin/python")
|
||||
ISSUE_DIR_RE = re.compile(r"^issue_(\d+)$")
|
||||
SIGNED_INTEGER_RE = re.compile(r"^[+-]?\d+$")
|
||||
FRAME_ID_CAMERA4_RE = re.compile(r"camera4\s*:\s*([+-]?\d+)", re.IGNORECASE)
|
||||
FRAME_ID_ANY_CAMERA_RE = re.compile(r"(camera\d+)\s*:\s*([+-]?\d+)", re.IGNORECASE)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class TargetFrame:
|
||||
camera: str
|
||||
frame_id: int
|
||||
raw_text: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class InferenceCase:
|
||||
issue_id: int
|
||||
issue_dir: Path
|
||||
case_dir: Path
|
||||
camera4_bin: Path
|
||||
relative_case_dir: Path
|
||||
output_dir: Path
|
||||
frame_window_source: str
|
||||
target_frame_text: str | None = None
|
||||
target_frame_id: int | None = None
|
||||
requested_frame_index_start: int | None = None
|
||||
requested_frame_index_end: int | None = None
|
||||
requested_frame_id_start: int | None = None
|
||||
requested_frame_id_end: int | None = None
|
||||
|
||||
|
||||
@dataclass
|
||||
class CaseResult:
|
||||
issue_id: int
|
||||
issue_dir: str
|
||||
case_dir: str
|
||||
camera4_bin: str
|
||||
relative_case_dir: str
|
||||
output_dir: str
|
||||
status: str
|
||||
detail: str
|
||||
command: list[str]
|
||||
log_path: str | None = None
|
||||
frame_window_source: str | None = None
|
||||
target_frame_text: str | None = None
|
||||
target_frame_id: int | None = None
|
||||
requested_frame_index_start: int | None = None
|
||||
requested_frame_index_end: int | None = None
|
||||
requested_frame_id_start: int | None = None
|
||||
requested_frame_id_end: int | None = None
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
return {
|
||||
"issue_id": self.issue_id,
|
||||
"issue_dir": self.issue_dir,
|
||||
"case_dir": self.case_dir,
|
||||
"camera4_bin": self.camera4_bin,
|
||||
"relative_case_dir": self.relative_case_dir,
|
||||
"output_dir": self.output_dir,
|
||||
"status": self.status,
|
||||
"detail": self.detail,
|
||||
"command": self.command,
|
||||
"log_path": self.log_path,
|
||||
"frame_window_source": self.frame_window_source,
|
||||
"target_frame_text": self.target_frame_text,
|
||||
"target_frame_id": self.target_frame_id,
|
||||
"requested_frame_index_start": self.requested_frame_index_start,
|
||||
"requested_frame_index_end": self.requested_frame_index_end,
|
||||
"requested_frame_id_start": self.requested_frame_id_start,
|
||||
"requested_frame_id_end": self.requested_frame_id_end,
|
||||
}
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Run exported-model inference on all downloaded issue-data camera4.bin cases."
|
||||
)
|
||||
parser.add_argument(
|
||||
"--download-root",
|
||||
default=str(DEFAULT_DOWNLOAD_ROOT),
|
||||
help="Root directory produced by download_issue_data.py.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--output-root",
|
||||
default=str(DEFAULT_OUTPUT_ROOT),
|
||||
help="Root directory where inference outputs will be written.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--manifest-path",
|
||||
default="",
|
||||
help="Optional explicit manifest JSON path. Defaults to <output-root>/inference_manifest.json",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--python-bin",
|
||||
default=str(DEFAULT_PYTHON_BIN),
|
||||
help="Python interpreter used to launch run_two_roi_exported_onnx_infer.py.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--inference-script",
|
||||
default=str(DEFAULT_INFERENCE_SCRIPT),
|
||||
help="Path to run_two_roi_exported_onnx_infer.py.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--issue-json",
|
||||
default="",
|
||||
help="Optional Feishu issue export JSON used to resolve 问题发生frameid windows.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--issue-id",
|
||||
action="append",
|
||||
dest="issue_ids",
|
||||
type=int,
|
||||
help="Optional issue id filter. Can be repeated.",
|
||||
)
|
||||
parser.add_argument("--video-stride", type=int, default=1)
|
||||
parser.add_argument("--max-images", type=int, default=0)
|
||||
parser.add_argument("--frame-index-start", type=int, default=None)
|
||||
parser.add_argument("--frame-index-end", type=int, default=None)
|
||||
parser.add_argument("--frame-id-start", type=int, default=None)
|
||||
parser.add_argument("--frame-id-end", type=int, default=None)
|
||||
parser.add_argument("--target-frame-id", type=int, default=None)
|
||||
parser.add_argument("--frame-before", type=int, default=100)
|
||||
parser.add_argument("--frame-after", type=int, default=100)
|
||||
parser.add_argument("--use-issue-frame-window", action="store_true")
|
||||
parser.add_argument(
|
||||
"--missing-issue-frame-policy",
|
||||
choices=("full", "skip"),
|
||||
default="full",
|
||||
help="How to handle cases without a usable 问题发生frameid when --use-issue-frame-window is enabled.",
|
||||
)
|
||||
parser.add_argument("--exported-model", type=str, default="")
|
||||
parser.add_argument("--device", type=str, default="")
|
||||
parser.add_argument("--providers", nargs="*", default=None)
|
||||
parser.add_argument("--enable-attr", action="store_true")
|
||||
parser.add_argument("--enable-cross-class-merge-prior", action="store_true")
|
||||
parser.add_argument("--save-aggregate-predictions", action="store_true")
|
||||
parser.add_argument(
|
||||
"--inference-arg",
|
||||
action="append",
|
||||
default=[],
|
||||
help="Extra argument forwarded to run_two_roi_exported_onnx_infer.py. Can be repeated.",
|
||||
)
|
||||
parser.add_argument("--skip-existing", action="store_true")
|
||||
parser.add_argument("--dry-run", action="store_true")
|
||||
args = parser.parse_args()
|
||||
if args.frame_before < 0:
|
||||
parser.error("--frame-before must be greater than or equal to 0")
|
||||
if args.frame_after < 0:
|
||||
parser.error("--frame-after must be greater than or equal to 0")
|
||||
if (
|
||||
args.frame_index_start is not None
|
||||
and args.frame_index_end is not None
|
||||
and args.frame_index_start > args.frame_index_end
|
||||
):
|
||||
parser.error("--frame-index-start must be less than or equal to --frame-index-end")
|
||||
if (
|
||||
args.frame_id_start is not None
|
||||
and args.frame_id_end is not None
|
||||
and args.frame_id_start > args.frame_id_end
|
||||
):
|
||||
parser.error("--frame-id-start must be less than or equal to --frame-id-end")
|
||||
if args.use_issue_frame_window and not args.issue_json:
|
||||
parser.error("--issue-json is required when --use-issue-frame-window is enabled")
|
||||
return args
|
||||
|
||||
|
||||
def ensure_dir(path: Path, dry_run: bool) -> None:
|
||||
if dry_run:
|
||||
return
|
||||
path.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
|
||||
def log_progress(message: str) -> None:
|
||||
timestamp = datetime.now().astimezone().strftime("%Y-%m-%d %H:%M:%S")
|
||||
print(f"[run_issue_data_inference {timestamp}] {message}", flush=True)
|
||||
|
||||
|
||||
def compact_text(value: object, max_len: int = 96) -> str:
|
||||
text = "" if value is None else str(value).strip()
|
||||
text = re.sub(r"\s+", " ", text)
|
||||
if len(text) <= max_len:
|
||||
return text
|
||||
return f"{text[: max_len - 3]}..."
|
||||
|
||||
|
||||
def parse_issue_id_from_path(path: Path) -> int | None:
|
||||
for part in path.parts:
|
||||
match = ISSUE_DIR_RE.match(part)
|
||||
if match:
|
||||
return int(match.group(1))
|
||||
return None
|
||||
|
||||
|
||||
def load_issue_items(path: Path) -> list[dict]:
|
||||
payload = json.loads(path.read_text(encoding="utf-8"))
|
||||
return payload["items"]
|
||||
|
||||
|
||||
def parse_target_frame(frame_text: object) -> tuple[TargetFrame | None, str]:
|
||||
text = "" if frame_text is None else str(frame_text).strip()
|
||||
if not text:
|
||||
return None, "missing 问题发生frameid"
|
||||
camera4_match = FRAME_ID_CAMERA4_RE.search(text)
|
||||
if camera4_match:
|
||||
frame_id = int(camera4_match.group(1))
|
||||
if frame_id <= 0:
|
||||
return None, f"non-positive 问题发生frameid: {text!r}"
|
||||
return TargetFrame(camera="camera4", frame_id=frame_id, raw_text=text), ""
|
||||
if SIGNED_INTEGER_RE.fullmatch(text):
|
||||
frame_id = int(text)
|
||||
if frame_id <= 0:
|
||||
return None, f"non-positive 问题发生frameid: {text!r}"
|
||||
return TargetFrame(camera="any", frame_id=frame_id, raw_text=text), ""
|
||||
any_camera_match = FRAME_ID_ANY_CAMERA_RE.search(text)
|
||||
if any_camera_match:
|
||||
frame_id = int(any_camera_match.group(2))
|
||||
if frame_id <= 0:
|
||||
return None, f"non-positive 问题发生frameid: {text!r}"
|
||||
return TargetFrame(camera=any_camera_match.group(1).lower(), frame_id=frame_id, raw_text=text), ""
|
||||
return None, f"unparseable 问题发生frameid: {text!r}"
|
||||
|
||||
|
||||
def build_issue_item_lookup(issue_json: Path) -> dict[int, dict]:
|
||||
return {int(item["id"]): item for item in load_issue_items(issue_json)}
|
||||
|
||||
|
||||
def has_manual_frame_window(args: argparse.Namespace) -> bool:
|
||||
return any(
|
||||
value is not None
|
||||
for value in (
|
||||
args.frame_index_start,
|
||||
args.frame_index_end,
|
||||
args.frame_id_start,
|
||||
args.frame_id_end,
|
||||
args.target_frame_id,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def build_case_result(
|
||||
case: InferenceCase,
|
||||
*,
|
||||
status: str,
|
||||
detail: str,
|
||||
command: list[str] | None = None,
|
||||
log_path: str | None = None,
|
||||
) -> CaseResult:
|
||||
return CaseResult(
|
||||
issue_id=case.issue_id,
|
||||
issue_dir=str(case.issue_dir),
|
||||
case_dir=str(case.case_dir),
|
||||
camera4_bin=str(case.camera4_bin),
|
||||
relative_case_dir=str(case.relative_case_dir),
|
||||
output_dir=str(case.output_dir),
|
||||
status=status,
|
||||
detail=detail,
|
||||
command=command or [],
|
||||
log_path=log_path,
|
||||
frame_window_source=case.frame_window_source,
|
||||
target_frame_text=case.target_frame_text,
|
||||
target_frame_id=case.target_frame_id,
|
||||
requested_frame_index_start=case.requested_frame_index_start,
|
||||
requested_frame_index_end=case.requested_frame_index_end,
|
||||
requested_frame_id_start=case.requested_frame_id_start,
|
||||
requested_frame_id_end=case.requested_frame_id_end,
|
||||
)
|
||||
|
||||
|
||||
def discover_cases(download_root: Path, output_root: Path, issue_filter: set[int] | None) -> list[InferenceCase]:
|
||||
cases: list[InferenceCase] = []
|
||||
for camera4_bin in sorted(download_root.rglob("camera4.bin")):
|
||||
if camera4_bin.parent.name != "sigmastar.1":
|
||||
continue
|
||||
try:
|
||||
relative_camera = camera4_bin.relative_to(download_root)
|
||||
except ValueError:
|
||||
continue
|
||||
|
||||
issue_id = parse_issue_id_from_path(relative_camera)
|
||||
if issue_id is None:
|
||||
continue
|
||||
if issue_filter and issue_id not in issue_filter:
|
||||
continue
|
||||
|
||||
case_dir = camera4_bin.parent.parent
|
||||
relative_case_dir = case_dir.relative_to(download_root)
|
||||
issue_dir = download_root / f"issue_{issue_id}"
|
||||
output_dir = output_root / relative_case_dir
|
||||
cases.append(
|
||||
InferenceCase(
|
||||
issue_id=issue_id,
|
||||
issue_dir=issue_dir,
|
||||
case_dir=case_dir,
|
||||
camera4_bin=camera4_bin,
|
||||
relative_case_dir=relative_case_dir,
|
||||
output_dir=output_dir,
|
||||
frame_window_source="full",
|
||||
)
|
||||
)
|
||||
return cases
|
||||
|
||||
|
||||
def apply_frame_windows(
|
||||
cases: list[InferenceCase],
|
||||
args: argparse.Namespace,
|
||||
issue_item_lookup: dict[int, dict],
|
||||
) -> tuple[list[InferenceCase], list[CaseResult]]:
|
||||
prepared_cases: list[InferenceCase] = []
|
||||
skipped_results: list[CaseResult] = []
|
||||
|
||||
for case in cases:
|
||||
prepared_case = InferenceCase(
|
||||
issue_id=case.issue_id,
|
||||
issue_dir=case.issue_dir,
|
||||
case_dir=case.case_dir,
|
||||
camera4_bin=case.camera4_bin,
|
||||
relative_case_dir=case.relative_case_dir,
|
||||
output_dir=case.output_dir,
|
||||
frame_window_source="full",
|
||||
)
|
||||
|
||||
if has_manual_frame_window(args):
|
||||
requested_frame_id_start = args.frame_id_start
|
||||
requested_frame_id_end = args.frame_id_end
|
||||
if args.target_frame_id is not None:
|
||||
if requested_frame_id_start is None:
|
||||
requested_frame_id_start = max(0, args.target_frame_id - args.frame_before)
|
||||
if requested_frame_id_end is None:
|
||||
requested_frame_id_end = args.target_frame_id + args.frame_after
|
||||
prepared_case = InferenceCase(
|
||||
issue_id=case.issue_id,
|
||||
issue_dir=case.issue_dir,
|
||||
case_dir=case.case_dir,
|
||||
camera4_bin=case.camera4_bin,
|
||||
relative_case_dir=case.relative_case_dir,
|
||||
output_dir=case.output_dir,
|
||||
frame_window_source="manual",
|
||||
target_frame_id=args.target_frame_id,
|
||||
requested_frame_index_start=args.frame_index_start,
|
||||
requested_frame_index_end=args.frame_index_end,
|
||||
requested_frame_id_start=requested_frame_id_start,
|
||||
requested_frame_id_end=requested_frame_id_end,
|
||||
)
|
||||
prepared_cases.append(prepared_case)
|
||||
continue
|
||||
|
||||
if not args.use_issue_frame_window:
|
||||
prepared_cases.append(prepared_case)
|
||||
continue
|
||||
|
||||
item = issue_item_lookup.get(case.issue_id)
|
||||
if item is None:
|
||||
if args.missing_issue_frame_policy == "skip":
|
||||
skipped_results.append(
|
||||
build_case_result(
|
||||
prepared_case,
|
||||
status="skipped_missing_issue_record",
|
||||
detail="issue id not found in issue_json",
|
||||
)
|
||||
)
|
||||
continue
|
||||
prepared_case = InferenceCase(
|
||||
issue_id=case.issue_id,
|
||||
issue_dir=case.issue_dir,
|
||||
case_dir=case.case_dir,
|
||||
camera4_bin=case.camera4_bin,
|
||||
relative_case_dir=case.relative_case_dir,
|
||||
output_dir=case.output_dir,
|
||||
frame_window_source="full_missing_issue_record",
|
||||
)
|
||||
prepared_cases.append(prepared_case)
|
||||
continue
|
||||
|
||||
target_frame, target_frame_error = parse_target_frame(item.get("问题发生frameid"))
|
||||
if target_frame is None:
|
||||
if args.missing_issue_frame_policy == "skip":
|
||||
skipped_results.append(
|
||||
build_case_result(
|
||||
prepared_case,
|
||||
status="skipped_missing_issue_frame",
|
||||
detail=target_frame_error,
|
||||
)
|
||||
)
|
||||
continue
|
||||
prepared_case = InferenceCase(
|
||||
issue_id=case.issue_id,
|
||||
issue_dir=case.issue_dir,
|
||||
case_dir=case.case_dir,
|
||||
camera4_bin=case.camera4_bin,
|
||||
relative_case_dir=case.relative_case_dir,
|
||||
output_dir=case.output_dir,
|
||||
frame_window_source=(
|
||||
"full_missing_issue_frame"
|
||||
if target_frame_error.startswith("missing ")
|
||||
else "full_invalid_issue_frame"
|
||||
),
|
||||
)
|
||||
prepared_cases.append(prepared_case)
|
||||
continue
|
||||
|
||||
if target_frame.camera not in {"camera4", "any"}:
|
||||
if args.missing_issue_frame_policy == "skip":
|
||||
skipped_results.append(
|
||||
build_case_result(
|
||||
prepared_case,
|
||||
status="skipped_unsupported_issue_camera",
|
||||
detail=f"unsupported target camera for window inference: {target_frame.camera}",
|
||||
)
|
||||
)
|
||||
continue
|
||||
prepared_case = InferenceCase(
|
||||
issue_id=case.issue_id,
|
||||
issue_dir=case.issue_dir,
|
||||
case_dir=case.case_dir,
|
||||
camera4_bin=case.camera4_bin,
|
||||
relative_case_dir=case.relative_case_dir,
|
||||
output_dir=case.output_dir,
|
||||
frame_window_source="full_unsupported_issue_camera",
|
||||
target_frame_text=target_frame.raw_text,
|
||||
)
|
||||
prepared_cases.append(prepared_case)
|
||||
continue
|
||||
|
||||
prepared_case = InferenceCase(
|
||||
issue_id=case.issue_id,
|
||||
issue_dir=case.issue_dir,
|
||||
case_dir=case.case_dir,
|
||||
camera4_bin=case.camera4_bin,
|
||||
relative_case_dir=case.relative_case_dir,
|
||||
output_dir=case.output_dir,
|
||||
frame_window_source="issue_frame",
|
||||
target_frame_text=target_frame.raw_text,
|
||||
target_frame_id=target_frame.frame_id,
|
||||
requested_frame_id_start=max(0, target_frame.frame_id - args.frame_before),
|
||||
requested_frame_id_end=target_frame.frame_id + args.frame_after,
|
||||
)
|
||||
prepared_cases.append(prepared_case)
|
||||
|
||||
return prepared_cases, skipped_results
|
||||
|
||||
|
||||
def has_existing_outputs(output_dir: Path) -> bool:
|
||||
if (output_dir / "predictions_merged.json").is_file():
|
||||
return True
|
||||
merge_json_dir = output_dir / "predictions" / "merge"
|
||||
if merge_json_dir.is_dir():
|
||||
for child in merge_json_dir.iterdir():
|
||||
if child.is_file():
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def build_command(case: InferenceCase, args: argparse.Namespace) -> list[str]:
|
||||
command = [
|
||||
str(Path(args.python_bin).resolve()),
|
||||
str(Path(args.inference_script).resolve()),
|
||||
"--video-case-dir",
|
||||
str(case.camera4_bin),
|
||||
"--output-dir",
|
||||
str(case.output_dir),
|
||||
"--video-stride",
|
||||
str(args.video_stride),
|
||||
]
|
||||
if args.max_images > 0:
|
||||
command.extend(["--max-images", str(args.max_images)])
|
||||
if case.target_frame_id is not None:
|
||||
command.extend(["--target-frame-id", str(case.target_frame_id)])
|
||||
if case.requested_frame_index_start is not None:
|
||||
command.extend(["--frame-index-start", str(case.requested_frame_index_start)])
|
||||
if case.requested_frame_index_end is not None:
|
||||
command.extend(["--frame-index-end", str(case.requested_frame_index_end)])
|
||||
if case.requested_frame_id_start is not None:
|
||||
command.extend(["--frame-id-start", str(case.requested_frame_id_start)])
|
||||
if case.requested_frame_id_end is not None:
|
||||
command.extend(["--frame-id-end", str(case.requested_frame_id_end)])
|
||||
if args.exported_model:
|
||||
command.extend(["--exported-model", args.exported_model])
|
||||
if args.device:
|
||||
command.extend(["--device", args.device])
|
||||
if args.providers:
|
||||
command.extend(["--providers", *args.providers])
|
||||
if args.enable_attr:
|
||||
command.append("--enable-attr")
|
||||
if args.enable_cross_class_merge_prior:
|
||||
command.append("--enable-cross-class-merge-prior")
|
||||
if args.save_aggregate_predictions:
|
||||
command.append("--save-aggregate-predictions")
|
||||
for extra_arg in args.inference_arg:
|
||||
command.append(extra_arg)
|
||||
return command
|
||||
|
||||
|
||||
def write_case_status_files(case: InferenceCase, command: list[str], log_text: str, dry_run: bool) -> str:
|
||||
status_dir = case.output_dir / "_status"
|
||||
if dry_run:
|
||||
return str(status_dir / "inference.log")
|
||||
ensure_dir(status_dir, dry_run=False)
|
||||
(status_dir / "command.txt").write_text(" ".join(command) + "\n", encoding="utf-8")
|
||||
log_path = status_dir / "inference.log"
|
||||
log_path.write_text(log_text, encoding="utf-8")
|
||||
return str(log_path)
|
||||
|
||||
|
||||
def summarize_process_output(completed: subprocess.CompletedProcess[str]) -> str:
|
||||
stdout = completed.stdout.strip()
|
||||
stderr = completed.stderr.strip()
|
||||
if completed.returncode == 0:
|
||||
if stdout:
|
||||
return stdout.splitlines()[-1]
|
||||
return "inference completed"
|
||||
if stderr:
|
||||
return stderr.splitlines()[-1]
|
||||
if stdout:
|
||||
return stdout.splitlines()[-1]
|
||||
return f"inference failed with return code {completed.returncode}"
|
||||
|
||||
|
||||
def run_case(case: InferenceCase, args: argparse.Namespace) -> CaseResult:
|
||||
command = build_command(case, args)
|
||||
|
||||
if args.skip_existing and has_existing_outputs(case.output_dir):
|
||||
return build_case_result(
|
||||
case,
|
||||
status="skipped_existing",
|
||||
detail="existing inference outputs found",
|
||||
command=command,
|
||||
log_path=None,
|
||||
)
|
||||
|
||||
if args.dry_run:
|
||||
return build_case_result(
|
||||
case,
|
||||
status="planned",
|
||||
detail="would run inference",
|
||||
command=command,
|
||||
log_path=str(case.output_dir / "_status" / "inference.log"),
|
||||
)
|
||||
|
||||
ensure_dir(case.output_dir, dry_run=False)
|
||||
completed = subprocess.run(
|
||||
command,
|
||||
cwd=str(ROOT),
|
||||
check=False,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
encoding="utf-8",
|
||||
)
|
||||
combined_log = "\n".join(
|
||||
part for part in (completed.stdout.strip(), completed.stderr.strip()) if part
|
||||
)
|
||||
log_path = write_case_status_files(case, command, combined_log + ("\n" if combined_log else ""), dry_run=False)
|
||||
status = "success" if completed.returncode == 0 else "failed"
|
||||
detail = summarize_process_output(completed)
|
||||
return build_case_result(
|
||||
case,
|
||||
status=status,
|
||||
detail=detail,
|
||||
command=command,
|
||||
log_path=log_path,
|
||||
)
|
||||
|
||||
|
||||
def build_manifest(
|
||||
args: argparse.Namespace,
|
||||
download_root: Path,
|
||||
output_root: Path,
|
||||
discovered_cases: list[InferenceCase],
|
||||
results: list[CaseResult],
|
||||
) -> dict:
|
||||
summary: dict[str, int] = {}
|
||||
for result in results:
|
||||
summary[result.status] = summary.get(result.status, 0) + 1
|
||||
return {
|
||||
"generated_at": datetime.now().astimezone().isoformat(timespec="seconds"),
|
||||
"download_root": str(download_root),
|
||||
"output_root": str(output_root),
|
||||
"python_bin": str(Path(args.python_bin).resolve()),
|
||||
"inference_script": str(Path(args.inference_script).resolve()),
|
||||
"dry_run": args.dry_run,
|
||||
"skip_existing": args.skip_existing,
|
||||
"issue_filter": args.issue_ids or [],
|
||||
"issue_json": args.issue_json,
|
||||
"use_issue_frame_window": args.use_issue_frame_window,
|
||||
"missing_issue_frame_policy": args.missing_issue_frame_policy,
|
||||
"video_stride": args.video_stride,
|
||||
"max_images": args.max_images,
|
||||
"frame_index_start": args.frame_index_start,
|
||||
"frame_index_end": args.frame_index_end,
|
||||
"frame_id_start": args.frame_id_start,
|
||||
"frame_id_end": args.frame_id_end,
|
||||
"target_frame_id": args.target_frame_id,
|
||||
"frame_before": args.frame_before,
|
||||
"frame_after": args.frame_after,
|
||||
"exported_model": args.exported_model,
|
||||
"device": args.device,
|
||||
"providers": args.providers or [],
|
||||
"enable_attr": args.enable_attr,
|
||||
"enable_cross_class_merge_prior": args.enable_cross_class_merge_prior,
|
||||
"save_aggregate_predictions": args.save_aggregate_predictions,
|
||||
"inference_args": args.inference_arg,
|
||||
"total_cases": len(discovered_cases),
|
||||
"summary": summary,
|
||||
"cases": [result.to_dict() for result in results],
|
||||
}
|
||||
|
||||
|
||||
def print_summary(manifest: dict) -> None:
|
||||
print(f"download_root: {manifest['download_root']}")
|
||||
print(f"output_root: {manifest['output_root']}")
|
||||
print(f"dry_run: {manifest['dry_run']}")
|
||||
if manifest["use_issue_frame_window"]:
|
||||
print(
|
||||
"issue_frame_window: "
|
||||
f"enabled issue_json={manifest['issue_json']} "
|
||||
f"before={manifest['frame_before']} after={manifest['frame_after']} "
|
||||
f"missing_policy={manifest['missing_issue_frame_policy']}"
|
||||
)
|
||||
elif (
|
||||
manifest["target_frame_id"] is not None
|
||||
or manifest["frame_id_start"] is not None
|
||||
or manifest["frame_id_end"] is not None
|
||||
or manifest["frame_index_start"] is not None
|
||||
or manifest["frame_index_end"] is not None
|
||||
):
|
||||
print(
|
||||
"manual_frame_window: "
|
||||
f"target_frame_id={manifest['target_frame_id']} "
|
||||
f"frame_id=[{manifest['frame_id_start'] if manifest['frame_id_start'] is not None else '-inf'}, "
|
||||
f"{manifest['frame_id_end'] if manifest['frame_id_end'] is not None else '+inf'}] "
|
||||
f"frame_index=[{manifest['frame_index_start'] if manifest['frame_index_start'] is not None else '-inf'}, "
|
||||
f"{manifest['frame_index_end'] if manifest['frame_index_end'] is not None else '+inf'}]"
|
||||
)
|
||||
print(f"total_cases: {manifest['total_cases']}")
|
||||
for status, count in sorted(manifest["summary"].items()):
|
||||
print(f"{status}: {count}")
|
||||
|
||||
|
||||
def main() -> int:
|
||||
args = parse_args()
|
||||
download_root = Path(args.download_root).resolve()
|
||||
output_root = Path(args.output_root).resolve()
|
||||
manifest_path = (
|
||||
Path(args.manifest_path).resolve()
|
||||
if args.manifest_path
|
||||
else output_root / "inference_manifest.json"
|
||||
)
|
||||
|
||||
issue_filter = set(args.issue_ids) if args.issue_ids else None
|
||||
cases = discover_cases(download_root, output_root, issue_filter)
|
||||
if not cases:
|
||||
raise FileNotFoundError(f"No */sigmastar.1/camera4.bin cases found under {download_root}")
|
||||
|
||||
issue_item_lookup: dict[int, dict] = {}
|
||||
if args.use_issue_frame_window:
|
||||
issue_item_lookup = build_issue_item_lookup(Path(args.issue_json).resolve())
|
||||
|
||||
prepared_cases, skipped_results = apply_frame_windows(cases, args, issue_item_lookup)
|
||||
log_progress(
|
||||
"cases discovered: "
|
||||
f"total={len(cases)} runnable={len(prepared_cases)} skipped_precheck={len(skipped_results)}"
|
||||
)
|
||||
for skipped_result in skipped_results:
|
||||
log_progress(
|
||||
f"skip issue_{skipped_result.issue_id} {compact_text(skipped_result.relative_case_dir)}: "
|
||||
f"{skipped_result.status} ({compact_text(skipped_result.detail, max_len=72)})"
|
||||
)
|
||||
|
||||
ensure_dir(output_root, dry_run=args.dry_run)
|
||||
results = list(skipped_results)
|
||||
total_prepared_cases = len(prepared_cases)
|
||||
for index, case in enumerate(prepared_cases, start=1):
|
||||
log_progress(
|
||||
f"[{index}/{total_prepared_cases}] issue_{case.issue_id} "
|
||||
f"{compact_text(case.relative_case_dir)}: start (window={case.frame_window_source})"
|
||||
)
|
||||
case_result = run_case(case, args)
|
||||
results.append(case_result)
|
||||
log_progress(
|
||||
f"[{index}/{total_prepared_cases}] issue_{case.issue_id} "
|
||||
f"{compact_text(case.relative_case_dir)}: "
|
||||
f"{case_result.status} ({compact_text(case_result.detail, max_len=72)})"
|
||||
)
|
||||
manifest = build_manifest(args, download_root, output_root, cases, results)
|
||||
|
||||
if not args.dry_run:
|
||||
ensure_dir(manifest_path.parent, dry_run=False)
|
||||
manifest_path.write_text(
|
||||
json.dumps(manifest, ensure_ascii=False, indent=2) + "\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
print_summary(manifest)
|
||||
if args.dry_run:
|
||||
print(f"manifest (not written in dry-run): {manifest_path}")
|
||||
else:
|
||||
print(f"manifest: {manifest_path}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
193
tools/feishu_project/run_issue_data_inference.sh
Executable file
193
tools/feishu_project/run_issue_data_inference.sh
Executable file
@@ -0,0 +1,193 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
|
||||
PROJECT_ROOT=$(cd "${SCRIPT_DIR}/../.." && pwd)
|
||||
|
||||
DOWNLOAD_ROOT=${DOWNLOAD_ROOT:-/data1/dongying/Mono3d/G1Q3/feishu_project/downloaded_issue_data}
|
||||
OUTPUT_ROOT=${OUTPUT_ROOT:-/data1/dongying/Mono3d/G1Q3/feishu_project/inference_issue_data}
|
||||
MANIFEST_PATH=${MANIFEST_PATH:-"${OUTPUT_ROOT}/inference_manifest.json"}
|
||||
PYTHON_BIN=${PYTHON_BIN:-/deeplearning_team/ydong/dongying/miniconda/envs/dev/bin/python}
|
||||
INFERENCE_SCRIPT=${INFERENCE_SCRIPT:-"${PROJECT_ROOT}/tools/model_inference/core/run_two_roi_exported_onnx_infer.py"}
|
||||
ISSUE_TRACKING_SCRIPT=${ISSUE_TRACKING_SCRIPT:-"${PROJECT_ROOT}/tools/feishu_project/run_issue_data_tracking.sh"}
|
||||
ISSUE_JSON=${ISSUE_JSON:-}
|
||||
VIDEO_STRIDE=${VIDEO_STRIDE:-1}
|
||||
MAX_IMAGES=${MAX_IMAGES:-0}
|
||||
FRAME_INDEX_START=${FRAME_INDEX_START:-}
|
||||
FRAME_INDEX_END=${FRAME_INDEX_END:-}
|
||||
FRAME_ID_START=${FRAME_ID_START:-}
|
||||
FRAME_ID_END=${FRAME_ID_END:-}
|
||||
TARGET_FRAME_ID=${TARGET_FRAME_ID:-}
|
||||
FRAME_BEFORE=${FRAME_BEFORE:-100}
|
||||
FRAME_AFTER=${FRAME_AFTER:-100}
|
||||
USE_ISSUE_FRAME_WINDOW=${USE_ISSUE_FRAME_WINDOW:-0}
|
||||
MISSING_ISSUE_FRAME_POLICY=${MISSING_ISSUE_FRAME_POLICY:-full}
|
||||
EXPORTED_MODEL=${EXPORTED_MODEL:-${PROJECT_ROOT}/runs/export/train_mono3d_two_roi_20260416-raw_no_edge/merged_model.torchscript}
|
||||
DEVICE=${DEVICE:-}
|
||||
PROVIDERS=${PROVIDERS:-}
|
||||
ENABLE_ATTR=${ENABLE_ATTR:-0}
|
||||
ENABLE_CROSS_CLASS_MERGE_PRIOR=${ENABLE_CROSS_CLASS_MERGE_PRIOR:-1}
|
||||
SAVE_AGGREGATE_PREDICTIONS=${SAVE_AGGREGATE_PREDICTIONS:-1}
|
||||
SKIP_EXISTING=${SKIP_EXISTING:-0}
|
||||
DRY_RUN=${DRY_RUN:-0}
|
||||
ENABLE_TRACKING=${ENABLE_TRACKING:-0}
|
||||
INFERENCE_EXTRA_ARGS=${INFERENCE_EXTRA_ARGS:-}
|
||||
|
||||
resolve_tracking_model_version() {
|
||||
if [[ -n "${TRACKING_MODEL_VERSION:-}" ]]; then
|
||||
printf '%s\n' "${TRACKING_MODEL_VERSION}"
|
||||
return 0
|
||||
fi
|
||||
|
||||
if [[ -n "${EXPORTED_MODEL}" ]] && [[ "${EXPORTED_MODEL}" =~ ([0-9]{8}) ]]; then
|
||||
printf '%s\n' "${BASH_REMATCH[1]}"
|
||||
return 0
|
||||
fi
|
||||
|
||||
return 1
|
||||
}
|
||||
|
||||
FORWARD_ARGS=()
|
||||
ISSUE_ID_ARGS=()
|
||||
while (($# > 0)); do
|
||||
case "$1" in
|
||||
--issue-id)
|
||||
if (($# < 2)); then
|
||||
echo "Error: --issue-id requires a value" >&2
|
||||
exit 1
|
||||
fi
|
||||
ISSUE_ID_ARGS+=("$1" "$2")
|
||||
FORWARD_ARGS+=("$1" "$2")
|
||||
shift 2
|
||||
;;
|
||||
--issue-id=*)
|
||||
ISSUE_ID_ARGS+=("$1")
|
||||
FORWARD_ARGS+=("$1")
|
||||
shift
|
||||
;;
|
||||
*)
|
||||
FORWARD_ARGS+=("$1")
|
||||
shift
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
CMD=(
|
||||
"${PYTHON_BIN}" "${PROJECT_ROOT}/tools/feishu_project/run_issue_data_inference.py"
|
||||
--download-root "${DOWNLOAD_ROOT}"
|
||||
--output-root "${OUTPUT_ROOT}"
|
||||
--manifest-path "${MANIFEST_PATH}"
|
||||
--python-bin "${PYTHON_BIN}"
|
||||
--inference-script "${INFERENCE_SCRIPT}"
|
||||
--video-stride "${VIDEO_STRIDE}"
|
||||
)
|
||||
|
||||
if [[ -n "${ISSUE_JSON}" ]]; then
|
||||
CMD+=(--issue-json "${ISSUE_JSON}")
|
||||
fi
|
||||
|
||||
if [[ "${MAX_IMAGES}" != "0" ]]; then
|
||||
CMD+=(--max-images "${MAX_IMAGES}")
|
||||
fi
|
||||
|
||||
if [[ -n "${FRAME_INDEX_START}" ]]; then
|
||||
CMD+=(--frame-index-start "${FRAME_INDEX_START}")
|
||||
fi
|
||||
|
||||
if [[ -n "${FRAME_INDEX_END}" ]]; then
|
||||
CMD+=(--frame-index-end "${FRAME_INDEX_END}")
|
||||
fi
|
||||
|
||||
if [[ -n "${FRAME_ID_START}" ]]; then
|
||||
CMD+=(--frame-id-start "${FRAME_ID_START}")
|
||||
fi
|
||||
|
||||
if [[ -n "${FRAME_ID_END}" ]]; then
|
||||
CMD+=(--frame-id-end "${FRAME_ID_END}")
|
||||
fi
|
||||
|
||||
if [[ -n "${TARGET_FRAME_ID}" ]]; then
|
||||
CMD+=(--target-frame-id "${TARGET_FRAME_ID}")
|
||||
fi
|
||||
|
||||
if [[ "${USE_ISSUE_FRAME_WINDOW}" != "1" && "${FRAME_BEFORE}" != "100" ]]; then
|
||||
CMD+=(--frame-before "${FRAME_BEFORE}")
|
||||
fi
|
||||
|
||||
if [[ "${USE_ISSUE_FRAME_WINDOW}" != "1" && "${FRAME_AFTER}" != "100" ]]; then
|
||||
CMD+=(--frame-after "${FRAME_AFTER}")
|
||||
fi
|
||||
|
||||
if [[ "${USE_ISSUE_FRAME_WINDOW}" == "1" ]]; then
|
||||
CMD+=(--use-issue-frame-window --frame-before "${FRAME_BEFORE}" --frame-after "${FRAME_AFTER}" --missing-issue-frame-policy "${MISSING_ISSUE_FRAME_POLICY}")
|
||||
fi
|
||||
|
||||
if [[ -n "${EXPORTED_MODEL}" ]]; then
|
||||
CMD+=(--exported-model "${EXPORTED_MODEL}")
|
||||
fi
|
||||
|
||||
if [[ -n "${DEVICE}" ]]; then
|
||||
CMD+=(--device "${DEVICE}")
|
||||
fi
|
||||
|
||||
if [[ -n "${PROVIDERS}" ]]; then
|
||||
# shellcheck disable=SC2206
|
||||
PROVIDER_ARR=(${PROVIDERS})
|
||||
CMD+=(--providers "${PROVIDER_ARR[@]}")
|
||||
fi
|
||||
|
||||
if [[ "${ENABLE_ATTR}" == "1" ]]; then
|
||||
CMD+=(--enable-attr)
|
||||
fi
|
||||
|
||||
if [[ "${ENABLE_CROSS_CLASS_MERGE_PRIOR}" == "1" ]]; then
|
||||
CMD+=(--enable-cross-class-merge-prior)
|
||||
fi
|
||||
|
||||
if [[ "${SAVE_AGGREGATE_PREDICTIONS}" == "1" ]]; then
|
||||
CMD+=(--save-aggregate-predictions)
|
||||
fi
|
||||
|
||||
if [[ "${SKIP_EXISTING}" == "1" ]]; then
|
||||
CMD+=(--skip-existing)
|
||||
fi
|
||||
|
||||
if [[ "${DRY_RUN}" == "1" ]]; then
|
||||
CMD+=(--dry-run)
|
||||
fi
|
||||
|
||||
if [[ -n "${INFERENCE_EXTRA_ARGS}" ]]; then
|
||||
# shellcheck disable=SC2206
|
||||
EXTRA_ARR=(${INFERENCE_EXTRA_ARGS})
|
||||
for arg in "${EXTRA_ARR[@]}"; do
|
||||
CMD+=("--inference-arg=${arg}")
|
||||
done
|
||||
fi
|
||||
|
||||
CMD+=("${FORWARD_ARGS[@]}")
|
||||
"${CMD[@]}"
|
||||
|
||||
if [[ "${ENABLE_TRACKING}" != "1" ]]; then
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if [[ "${DRY_RUN}" == "1" ]]; then
|
||||
echo "Skipping tracking because DRY_RUN=1"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
TRACKING_MODEL_VERSION_RESOLVED=${TRACKING_MODEL_VERSION:-}
|
||||
if [[ -z "${TRACKING_MODEL_VERSION_RESOLVED}" ]]; then
|
||||
TRACKING_MODEL_VERSION_RESOLVED=$(resolve_tracking_model_version || true)
|
||||
fi
|
||||
|
||||
if [[ -n "${TRACKING_MODEL_VERSION_RESOLVED}" ]]; then
|
||||
echo "Running tracking with model version: ${TRACKING_MODEL_VERSION_RESOLVED}"
|
||||
TRACKING_MODEL_VERSION="${TRACKING_MODEL_VERSION_RESOLVED}" \
|
||||
PYTHON_BIN="${PYTHON_BIN}" \
|
||||
bash "${ISSUE_TRACKING_SCRIPT}" "${OUTPUT_ROOT}" "${ISSUE_ID_ARGS[@]}"
|
||||
else
|
||||
echo "Running tracking without an explicit model version override"
|
||||
PYTHON_BIN="${PYTHON_BIN}" \
|
||||
bash "${ISSUE_TRACKING_SCRIPT}" "${OUTPUT_ROOT}" "${ISSUE_ID_ARGS[@]}"
|
||||
fi
|
||||
360
tools/feishu_project/run_issue_data_temporal_observe.sh
Executable file
360
tools/feishu_project/run_issue_data_temporal_observe.sh
Executable file
@@ -0,0 +1,360 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
|
||||
PROJECT_ROOT=$(cd "${SCRIPT_DIR}/../.." && pwd)
|
||||
export PYTHONPATH="${PROJECT_ROOT}:${PYTHONPATH:-}"
|
||||
|
||||
RESULTS_ROOT=${RESULTS_ROOT:-/data1/dongying/Mono3d/G1Q3/feishu_project/inference_issue_data}
|
||||
OUTPUT_ROOT=${OUTPUT_ROOT:-}
|
||||
OUTPUT_DIR_NAME=${OUTPUT_DIR_NAME:-temporal_observation}
|
||||
PYTHON_BIN=${PYTHON_BIN:-/deeplearning_team/ydong/dongying/miniconda/envs/dev/bin/python}
|
||||
TEMPORAL_SCRIPT=${TEMPORAL_SCRIPT:-"${PROJECT_ROOT}/tools/temporal_analysis/evaluate_temporal_stability.py"}
|
||||
TRACKING_JSON_NAME=${TRACKING_JSON_NAME:-merge.json}
|
||||
TEMPORAL_MIN_LENGTH=${TEMPORAL_MIN_LENGTH:-3}
|
||||
TEMPORAL_CLASS_ID=${TEMPORAL_CLASS_ID:-}
|
||||
TEMPORAL_TRACK_IDS=${TEMPORAL_TRACK_IDS:-4}
|
||||
TEMPORAL_FRAME_ID_START=${TEMPORAL_FRAME_ID_START:-86421}
|
||||
TEMPORAL_FRAME_ID_END=${TEMPORAL_FRAME_ID_END:-86441}
|
||||
TEMPORAL_PLOTS=${TEMPORAL_PLOTS:-0}
|
||||
TEMPORAL_EXPORT_SERIES=${TEMPORAL_EXPORT_SERIES:-1}
|
||||
TEMPORAL_FOCUS_TRACK_PLOTS=${TEMPORAL_FOCUS_TRACK_PLOTS:-1}
|
||||
TEMPORAL_PREFER_EGO=${TEMPORAL_PREFER_EGO:-1}
|
||||
TEMPORAL_X_AXIS=${TEMPORAL_X_AXIS:-frame_id}
|
||||
TEMPORAL_HEADING_SOURCE=${TEMPORAL_HEADING_SOURCE:-camera_reg}
|
||||
|
||||
TARGET_PATH="/data1/dongying/Mono3d/G1Q3/feishu_project/inference_issue_data/issue_6974203002/pdcl_01/20260417112730/2026-04-17_11-29-29_voice-HMI二轮车位置不正确"
|
||||
ISSUE_IDS=()
|
||||
CLI_TRACK_IDS=()
|
||||
CLI_CLASS_ID=""
|
||||
CLI_FRAME_ID_START=""
|
||||
CLI_FRAME_ID_END=""
|
||||
|
||||
while (($# > 0)); do
|
||||
case "$1" in
|
||||
--issue-id)
|
||||
if (($# < 2)); then
|
||||
echo "Error: --issue-id requires a value" >&2
|
||||
exit 1
|
||||
fi
|
||||
ISSUE_IDS+=("$2")
|
||||
shift 2
|
||||
;;
|
||||
--issue-id=*)
|
||||
ISSUE_IDS+=("${1#*=}")
|
||||
shift
|
||||
;;
|
||||
--track-id)
|
||||
if (($# < 2)); then
|
||||
echo "Error: --track-id requires a value" >&2
|
||||
exit 1
|
||||
fi
|
||||
CLI_TRACK_IDS+=("$2")
|
||||
shift 2
|
||||
;;
|
||||
--track-id=*)
|
||||
CLI_TRACK_IDS+=("${1#*=}")
|
||||
shift
|
||||
;;
|
||||
--class-id)
|
||||
if (($# < 2)); then
|
||||
echo "Error: --class-id requires a value" >&2
|
||||
exit 1
|
||||
fi
|
||||
CLI_CLASS_ID="$2"
|
||||
shift 2
|
||||
;;
|
||||
--class-id=*)
|
||||
CLI_CLASS_ID="${1#*=}"
|
||||
shift
|
||||
;;
|
||||
--frame-id-start)
|
||||
if (($# < 2)); then
|
||||
echo "Error: --frame-id-start requires a value" >&2
|
||||
exit 1
|
||||
fi
|
||||
CLI_FRAME_ID_START="$2"
|
||||
shift 2
|
||||
;;
|
||||
--frame-id-start=*)
|
||||
CLI_FRAME_ID_START="${1#*=}"
|
||||
shift
|
||||
;;
|
||||
--frame-id-end)
|
||||
if (($# < 2)); then
|
||||
echo "Error: --frame-id-end requires a value" >&2
|
||||
exit 1
|
||||
fi
|
||||
CLI_FRAME_ID_END="$2"
|
||||
shift 2
|
||||
;;
|
||||
--frame-id-end=*)
|
||||
CLI_FRAME_ID_END="${1#*=}"
|
||||
shift
|
||||
;;
|
||||
-*)
|
||||
echo "Error: unsupported option: $1" >&2
|
||||
exit 1
|
||||
;;
|
||||
*)
|
||||
if [[ -n "${TARGET_PATH}" ]]; then
|
||||
echo "Error: multiple target paths provided: ${TARGET_PATH} and $1" >&2
|
||||
exit 1
|
||||
fi
|
||||
TARGET_PATH="$1"
|
||||
shift
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
if [[ -z "${TARGET_PATH}" ]]; then
|
||||
TARGET_PATH="${RESULTS_ROOT}"
|
||||
fi
|
||||
|
||||
if [[ ! -e "${TARGET_PATH}" ]]; then
|
||||
echo "Error: target path does not exist: ${TARGET_PATH}" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
resolve_abs_path() {
|
||||
local target_path="$1"
|
||||
if [[ -d "${target_path}" ]]; then
|
||||
(
|
||||
cd "${target_path}"
|
||||
pwd -P
|
||||
)
|
||||
return 0
|
||||
fi
|
||||
|
||||
local parent_dir
|
||||
parent_dir=$(
|
||||
cd "$(dirname "${target_path}")"
|
||||
pwd -P
|
||||
)
|
||||
printf '%s/%s\n' "${parent_dir}" "$(basename "${target_path}")"
|
||||
}
|
||||
|
||||
RESULTS_ROOT_ABS="$(resolve_abs_path "${RESULTS_ROOT}")"
|
||||
|
||||
is_case_dir() {
|
||||
local dir_path="$1"
|
||||
[[ -d "${dir_path}" ]] && [[ -f "${dir_path}/${TRACKING_JSON_NAME}" ]]
|
||||
}
|
||||
|
||||
derive_output_dir() {
|
||||
local case_dir="$1"
|
||||
local case_abs
|
||||
local rel_case_dir
|
||||
|
||||
case_abs="$(resolve_abs_path "${case_dir}")"
|
||||
if [[ -n "${OUTPUT_ROOT}" ]]; then
|
||||
if [[ "${case_abs}" == "${RESULTS_ROOT_ABS}" ]]; then
|
||||
printf '%s\n' "${OUTPUT_ROOT}"
|
||||
return 0
|
||||
fi
|
||||
if [[ "${case_abs}" == "${RESULTS_ROOT_ABS}"/* ]]; then
|
||||
rel_case_dir="${case_abs#"${RESULTS_ROOT_ABS}/"}"
|
||||
printf '%s/%s\n' "${OUTPUT_ROOT%/}" "${rel_case_dir}"
|
||||
return 0
|
||||
fi
|
||||
printf '%s/%s\n' "${OUTPUT_ROOT%/}" "$(basename "${case_abs}")"
|
||||
return 0
|
||||
fi
|
||||
|
||||
printf '%s/%s\n' "${case_abs}" "${OUTPUT_DIR_NAME}"
|
||||
}
|
||||
|
||||
build_track_id_args() {
|
||||
local -n out_ref=$1
|
||||
out_ref=()
|
||||
|
||||
if [[ "${#CLI_TRACK_IDS[@]}" -gt 0 ]]; then
|
||||
for track_id in "${CLI_TRACK_IDS[@]}"; do
|
||||
out_ref+=(--track-ids "${track_id}")
|
||||
done
|
||||
return 0
|
||||
fi
|
||||
|
||||
if [[ -n "${TEMPORAL_TRACK_IDS}" ]]; then
|
||||
# shellcheck disable=SC2206
|
||||
local track_id_arr=(${TEMPORAL_TRACK_IDS})
|
||||
for track_id in "${track_id_arr[@]}"; do
|
||||
out_ref+=(--track-ids "${track_id}")
|
||||
done
|
||||
fi
|
||||
}
|
||||
|
||||
run_single_case() {
|
||||
local case_dir="$1"
|
||||
local tracking_json="${case_dir}/${TRACKING_JSON_NAME}"
|
||||
local output_dir
|
||||
local cmd
|
||||
local track_id_args
|
||||
local class_id_value=""
|
||||
local frame_id_start_value=""
|
||||
local frame_id_end_value=""
|
||||
|
||||
if [[ ! -f "${tracking_json}" ]]; then
|
||||
echo "[ERROR] ${TRACKING_JSON_NAME} not found in case directory: ${case_dir}" >&2
|
||||
return 1
|
||||
fi
|
||||
|
||||
output_dir="$(derive_output_dir "${case_dir}")"
|
||||
build_track_id_args track_id_args
|
||||
|
||||
if [[ -n "${CLI_CLASS_ID}" ]]; then
|
||||
class_id_value="${CLI_CLASS_ID}"
|
||||
elif [[ -n "${TEMPORAL_CLASS_ID}" ]]; then
|
||||
class_id_value="${TEMPORAL_CLASS_ID}"
|
||||
fi
|
||||
|
||||
if [[ -n "${CLI_FRAME_ID_START}" ]]; then
|
||||
frame_id_start_value="${CLI_FRAME_ID_START}"
|
||||
elif [[ -n "${TEMPORAL_FRAME_ID_START}" ]]; then
|
||||
frame_id_start_value="${TEMPORAL_FRAME_ID_START}"
|
||||
fi
|
||||
|
||||
if [[ -n "${CLI_FRAME_ID_END}" ]]; then
|
||||
frame_id_end_value="${CLI_FRAME_ID_END}"
|
||||
elif [[ -n "${TEMPORAL_FRAME_ID_END}" ]]; then
|
||||
frame_id_end_value="${TEMPORAL_FRAME_ID_END}"
|
||||
fi
|
||||
|
||||
cmd=(
|
||||
"${PYTHON_BIN}" "${TEMPORAL_SCRIPT}"
|
||||
--input "${tracking_json}"
|
||||
--output "${output_dir}/stability_report.json"
|
||||
--min-length "${TEMPORAL_MIN_LENGTH}"
|
||||
--x-axis "${TEMPORAL_X_AXIS}"
|
||||
--heading-source "${TEMPORAL_HEADING_SOURCE}"
|
||||
)
|
||||
|
||||
if [[ -n "${class_id_value}" ]]; then
|
||||
cmd+=(--class-id "${class_id_value}")
|
||||
fi
|
||||
|
||||
if [[ -n "${frame_id_start_value}" ]]; then
|
||||
cmd+=(--frame-id-start "${frame_id_start_value}")
|
||||
fi
|
||||
|
||||
if [[ -n "${frame_id_end_value}" ]]; then
|
||||
cmd+=(--frame-id-end "${frame_id_end_value}")
|
||||
fi
|
||||
|
||||
if [[ "${#track_id_args[@]}" -gt 0 ]]; then
|
||||
cmd+=("${track_id_args[@]}")
|
||||
fi
|
||||
|
||||
if [[ "${TEMPORAL_PLOTS}" == "1" ]]; then
|
||||
cmd+=(--plots)
|
||||
fi
|
||||
|
||||
if [[ "${TEMPORAL_EXPORT_SERIES}" == "1" ]]; then
|
||||
cmd+=(--export-series)
|
||||
fi
|
||||
|
||||
if [[ "${TEMPORAL_FOCUS_TRACK_PLOTS}" == "1" ]]; then
|
||||
cmd+=(--focus-track-plots)
|
||||
fi
|
||||
|
||||
if [[ "${TEMPORAL_PREFER_EGO}" == "1" ]]; then
|
||||
cmd+=(--prefer-ego)
|
||||
else
|
||||
cmd+=(--no-prefer-ego)
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "######################################################################"
|
||||
echo "# Feishu issue-data temporal observation"
|
||||
echo "######################################################################"
|
||||
echo "Case : ${case_dir}"
|
||||
echo "Track : ${tracking_json}"
|
||||
echo "Output : ${output_dir}"
|
||||
if [[ -n "${class_id_value}" ]]; then
|
||||
echo "Class : ${class_id_value}"
|
||||
fi
|
||||
if [[ -n "${frame_id_start_value}" || -n "${frame_id_end_value}" ]]; then
|
||||
echo "Frame ID Range: [${frame_id_start_value:-"-inf"}, ${frame_id_end_value:-"+inf"}]"
|
||||
fi
|
||||
echo "Heading Source: ${TEMPORAL_HEADING_SOURCE}"
|
||||
if [[ "${#CLI_TRACK_IDS[@]}" -gt 0 ]]; then
|
||||
echo "Track IDs: ${CLI_TRACK_IDS[*]}"
|
||||
elif [[ -n "${TEMPORAL_TRACK_IDS}" ]]; then
|
||||
echo "Track IDs: ${TEMPORAL_TRACK_IDS}"
|
||||
fi
|
||||
|
||||
"${cmd[@]}"
|
||||
}
|
||||
|
||||
run_batch_root() {
|
||||
local batch_root="$1"
|
||||
local total_cases=0
|
||||
local success_cases=0
|
||||
local failed_cases=0
|
||||
|
||||
while IFS= read -r -d '' tracking_json; do
|
||||
local case_dir
|
||||
case_dir="$(dirname "${tracking_json}")"
|
||||
((total_cases += 1))
|
||||
|
||||
if run_single_case "${case_dir}"; then
|
||||
((success_cases += 1))
|
||||
else
|
||||
((failed_cases += 1))
|
||||
printf '[FAIL] case=%s\n' "${case_dir}" >&2
|
||||
fi
|
||||
done < <(
|
||||
find "${batch_root}" -type f -name "${TRACKING_JSON_NAME}" -print0 | sort -z
|
||||
)
|
||||
|
||||
if [[ "${total_cases}" -eq 0 ]]; then
|
||||
echo "[ERROR] No ${TRACKING_JSON_NAME} files were found under: ${batch_root}" >&2
|
||||
return 1
|
||||
fi
|
||||
|
||||
echo ""
|
||||
printf '[DONE] cases=%d success=%d failed=%d\n' \
|
||||
"${total_cases}" "${success_cases}" "${failed_cases}"
|
||||
|
||||
[[ "${failed_cases}" -eq 0 ]]
|
||||
}
|
||||
|
||||
if [[ "${#ISSUE_IDS[@]}" -eq 0 ]]; then
|
||||
if is_case_dir "${TARGET_PATH}"; then
|
||||
run_single_case "${TARGET_PATH}"
|
||||
elif [[ -d "${TARGET_PATH}" ]]; then
|
||||
run_batch_root "${TARGET_PATH}"
|
||||
else
|
||||
echo "Error: unsupported target path: ${TARGET_PATH}" >&2
|
||||
exit 1
|
||||
fi
|
||||
exit 0
|
||||
fi
|
||||
|
||||
total_issues=0
|
||||
success_issues=0
|
||||
failed_issues=0
|
||||
|
||||
for issue_id in "${ISSUE_IDS[@]}"; do
|
||||
issue_root="${TARGET_PATH}/issue_${issue_id}"
|
||||
((total_issues += 1))
|
||||
|
||||
if [[ ! -d "${issue_root}" ]]; then
|
||||
echo "[FAIL] issue_${issue_id}: not found under ${TARGET_PATH}" >&2
|
||||
((failed_issues += 1))
|
||||
continue
|
||||
fi
|
||||
|
||||
if run_batch_root "${issue_root}"; then
|
||||
((success_issues += 1))
|
||||
else
|
||||
((failed_issues += 1))
|
||||
echo "[FAIL] issue_${issue_id}: temporal observation failed" >&2
|
||||
fi
|
||||
done
|
||||
|
||||
echo ""
|
||||
printf '[DONE] issues=%d success=%d failed=%d\n' \
|
||||
"${total_issues}" "${success_issues}" "${failed_issues}"
|
||||
|
||||
[[ "${failed_issues}" -eq 0 ]]
|
||||
347
tools/feishu_project/run_issue_data_tracking.sh
Executable file
347
tools/feishu_project/run_issue_data_tracking.sh
Executable file
@@ -0,0 +1,347 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
|
||||
PROJECT_ROOT=$(cd "${SCRIPT_DIR}/../.." && pwd)
|
||||
|
||||
RESULTS_ROOT=${RESULTS_ROOT:-/data1/dongying/Mono3d/G1Q3/feishu_project/inference_issue_data}
|
||||
PYTHON_BIN=${PYTHON_BIN:-/deeplearning_team/ydong/dongying/miniconda/envs/dev/bin/python}
|
||||
TRACKING_LAUNCHER=${TRACKING_LAUNCHER:-"${PROJECT_ROOT}/tools/temporal_analysis/track_objects_exported_onnx_infer_case.sh"}
|
||||
ISSUE_CONVERT_SCRIPT=${ISSUE_CONVERT_SCRIPT:-"${PROJECT_ROOT}/tools/feishu_project/run_issue_data_convert_tracking.sh"}
|
||||
ISSUE_VISUALIZE_SCRIPT=${ISSUE_VISUALIZE_SCRIPT:-"${PROJECT_ROOT}/tools/feishu_project/run_issue_data_visualize_tracking.sh"}
|
||||
ISSUE_TEMPORAL_SCRIPT=${ISSUE_TEMPORAL_SCRIPT:-"${PROJECT_ROOT}/tools/feishu_project/run_issue_data_temporal_observe.sh"}
|
||||
TRACK_CLASSES=${TRACK_CLASSES:-0 1 2 3 4 5 6 7 8 9 10 11 12 17 18 19}
|
||||
IOU_THRESH=${IOU_THRESH:-0.3}
|
||||
MAX_AGE=${MAX_AGE:-5}
|
||||
MIN_HITS=${MIN_HITS:-1}
|
||||
DIST_THRESH=${DIST_THRESH:-100}
|
||||
MAX_3D_DISTANCE=${MAX_3D_DISTANCE:-10.0}
|
||||
MAX_FRAMES=${MAX_FRAMES:-}
|
||||
MERGE_OUTPUT_NAME=${MERGE_OUTPUT_NAME:-combined_tracking.json}
|
||||
FILE_PATTERN=${FILE_PATTERN:-*.json}
|
||||
ENABLE_USE_3D=${ENABLE_USE_3D:-0}
|
||||
ENABLE_CONVERT=${ENABLE_CONVERT:-0}
|
||||
ENABLE_VISUALIZE=${ENABLE_VISUALIZE:-1}
|
||||
ENABLE_TEMPORAL_ANALYSIS=${ENABLE_TEMPORAL_ANALYSIS:-0}
|
||||
CONVERT_OUTPUT_DIR_NAME=${CONVERT_OUTPUT_DIR_NAME:-objectlist}
|
||||
CONVERT_TRACKING_JSON_NAME=${CONVERT_TRACKING_JSON_NAME:-merge.json}
|
||||
CONVERT_CAM_ID=${CONVERT_CAM_ID:-}
|
||||
VIS_DOWNLOAD_ROOT=${VIS_DOWNLOAD_ROOT:-/data1/dongying/Mono3d/G1Q3/feishu_project/downloaded_issue_data}
|
||||
VIS_OUTPUT_ROOT=${VIS_OUTPUT_ROOT:-}
|
||||
VIS_OUTPUT_DIR_NAME=${VIS_OUTPUT_DIR_NAME:-tracking_vis_raw}
|
||||
VIS_TRACKING_JSON_NAME=${VIS_TRACKING_JSON_NAME:-merge.json}
|
||||
VIS_MAX_FRAMES=${VIS_MAX_FRAMES:-}
|
||||
VIS_CLASS_ID=${VIS_CLASS_ID:-}
|
||||
VIS_TRACK_IDS=${VIS_TRACK_IDS:-}
|
||||
VIS_SHOW_TRAJECTORY=${VIS_SHOW_TRAJECTORY:-0}
|
||||
TEMPORAL_OUTPUT_ROOT=${TEMPORAL_OUTPUT_ROOT:-}
|
||||
TEMPORAL_OUTPUT_DIR_NAME=${TEMPORAL_OUTPUT_DIR_NAME:-temporal_observation}
|
||||
TEMPORAL_JSON_NAME=${TEMPORAL_JSON_NAME:-merge.json}
|
||||
TEMPORAL_MIN_LENGTH=${TEMPORAL_MIN_LENGTH:-3}
|
||||
TEMPORAL_TRACK_IDS=${TEMPORAL_TRACK_IDS:-}
|
||||
TEMPORAL_CLASS_ID=${TEMPORAL_CLASS_ID:-}
|
||||
TEMPORAL_FRAME_ID_START=${TEMPORAL_FRAME_ID_START:-}
|
||||
TEMPORAL_FRAME_ID_END=${TEMPORAL_FRAME_ID_END:-}
|
||||
TEMPORAL_PLOTS=${TEMPORAL_PLOTS:-0}
|
||||
TEMPORAL_EXPORT_SERIES=${TEMPORAL_EXPORT_SERIES:-1}
|
||||
TEMPORAL_FOCUS_TRACK_PLOTS=${TEMPORAL_FOCUS_TRACK_PLOTS:-1}
|
||||
TEMPORAL_PREFER_EGO=${TEMPORAL_PREFER_EGO:-1}
|
||||
TEMPORAL_X_AXIS=${TEMPORAL_X_AXIS:-frame_id}
|
||||
TEMPORAL_HEADING_SOURCE=${TEMPORAL_HEADING_SOURCE:-camera_reg}
|
||||
TRACKING_MODEL_VERSION=${TRACKING_MODEL_VERSION:-20260416}
|
||||
|
||||
TARGET_PATH=${TARGET_PATH:-}
|
||||
ISSUE_IDS=()
|
||||
|
||||
while (($# > 0)); do
|
||||
case "$1" in
|
||||
--issue-id)
|
||||
if (($# < 2)); then
|
||||
echo "Error: --issue-id requires a value" >&2
|
||||
exit 1
|
||||
fi
|
||||
ISSUE_IDS+=("$2")
|
||||
shift 2
|
||||
;;
|
||||
--issue-id=*)
|
||||
ISSUE_IDS+=("${1#*=}")
|
||||
shift
|
||||
;;
|
||||
-*)
|
||||
echo "Error: unsupported option: $1" >&2
|
||||
exit 1
|
||||
;;
|
||||
*)
|
||||
if [[ -n "${TARGET_PATH}" ]]; then
|
||||
echo "Error: multiple target paths provided: ${TARGET_PATH} and $1" >&2
|
||||
exit 1
|
||||
fi
|
||||
TARGET_PATH="$1"
|
||||
shift
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
if [[ -z "${TARGET_PATH}" ]]; then
|
||||
TARGET_PATH="${RESULTS_ROOT}"
|
||||
fi
|
||||
|
||||
if [[ ! -d "${TARGET_PATH}" ]]; then
|
||||
echo "Error: target directory does not exist: ${TARGET_PATH}" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
run_tracking_wrapper() {
|
||||
local target_path="$1"
|
||||
|
||||
echo ""
|
||||
echo "######################################################################"
|
||||
echo "# Feishu issue-data tracking"
|
||||
echo "######################################################################"
|
||||
echo "Target path: ${target_path}"
|
||||
if [[ -n "${TRACKING_MODEL_VERSION}" ]]; then
|
||||
echo "Model version: ${TRACKING_MODEL_VERSION}"
|
||||
fi
|
||||
|
||||
if [[ -n "${TRACKING_MODEL_VERSION}" ]]; then
|
||||
MODEL_VERSION="${TRACKING_MODEL_VERSION}" \
|
||||
PYTHON_BIN="${PYTHON_BIN}" \
|
||||
RESULTS_ROOT="${target_path}" \
|
||||
TRACK_CLASSES="${TRACK_CLASSES}" \
|
||||
IOU_THRESH="${IOU_THRESH}" \
|
||||
MAX_AGE="${MAX_AGE}" \
|
||||
MIN_HITS="${MIN_HITS}" \
|
||||
DIST_THRESH="${DIST_THRESH}" \
|
||||
MAX_3D_DISTANCE="${MAX_3D_DISTANCE}" \
|
||||
MAX_FRAMES="${MAX_FRAMES}" \
|
||||
MERGE_OUTPUT_NAME="${MERGE_OUTPUT_NAME}" \
|
||||
FILE_PATTERN="${FILE_PATTERN}" \
|
||||
ENABLE_USE_3D="${ENABLE_USE_3D}" \
|
||||
bash "${TRACKING_LAUNCHER}" "${target_path}"
|
||||
return
|
||||
fi
|
||||
|
||||
PYTHON_BIN="${PYTHON_BIN}" \
|
||||
RESULTS_ROOT="${target_path}" \
|
||||
TRACK_CLASSES="${TRACK_CLASSES}" \
|
||||
IOU_THRESH="${IOU_THRESH}" \
|
||||
MAX_AGE="${MAX_AGE}" \
|
||||
MIN_HITS="${MIN_HITS}" \
|
||||
DIST_THRESH="${DIST_THRESH}" \
|
||||
MAX_3D_DISTANCE="${MAX_3D_DISTANCE}" \
|
||||
MAX_FRAMES="${MAX_FRAMES}" \
|
||||
MERGE_OUTPUT_NAME="${MERGE_OUTPUT_NAME}" \
|
||||
FILE_PATTERN="${FILE_PATTERN}" \
|
||||
ENABLE_USE_3D="${ENABLE_USE_3D}" \
|
||||
bash "${TRACKING_LAUNCHER}" "${target_path}"
|
||||
}
|
||||
|
||||
run_conversion_wrapper() {
|
||||
local target_path="$1"
|
||||
local convert_args=("${target_path}")
|
||||
|
||||
for issue_id in "${ISSUE_IDS[@]}"; do
|
||||
convert_args+=(--issue-id "${issue_id}")
|
||||
done
|
||||
|
||||
echo ""
|
||||
echo "######################################################################"
|
||||
echo "# Feishu issue-data protocol conversion"
|
||||
echo "######################################################################"
|
||||
echo "Target path : ${target_path}"
|
||||
echo "Tracking JSON : ${CONVERT_TRACKING_JSON_NAME}"
|
||||
echo "Output dir name : ${CONVERT_OUTPUT_DIR_NAME}"
|
||||
|
||||
if [[ -n "${CONVERT_CAM_ID}" ]]; then
|
||||
echo "Camera ID override: ${CONVERT_CAM_ID}"
|
||||
fi
|
||||
|
||||
PYTHON_BIN="${PYTHON_BIN}" \
|
||||
RESULTS_ROOT="${RESULTS_ROOT}" \
|
||||
OUTPUT_DIR_NAME="${CONVERT_OUTPUT_DIR_NAME}" \
|
||||
MERGE_JSON_NAME="${CONVERT_TRACKING_JSON_NAME}" \
|
||||
CAM_ID="${CONVERT_CAM_ID}" \
|
||||
bash "${ISSUE_CONVERT_SCRIPT}" "${convert_args[@]}"
|
||||
}
|
||||
|
||||
run_visualization_wrapper() {
|
||||
local target_path="$1"
|
||||
local visualize_args=("${target_path}")
|
||||
|
||||
for issue_id in "${ISSUE_IDS[@]}"; do
|
||||
visualize_args+=(--issue-id "${issue_id}")
|
||||
done
|
||||
|
||||
echo ""
|
||||
echo "######################################################################"
|
||||
echo "# Feishu issue-data raw-image visualization"
|
||||
echo "######################################################################"
|
||||
echo "Target path : ${target_path}"
|
||||
echo "Download root : ${VIS_DOWNLOAD_ROOT}"
|
||||
if [[ -n "${VIS_OUTPUT_ROOT}" ]]; then
|
||||
echo "Output root : ${VIS_OUTPUT_ROOT}"
|
||||
else
|
||||
echo "Output dir name: ${VIS_OUTPUT_DIR_NAME}"
|
||||
fi
|
||||
|
||||
PYTHON_BIN="${PYTHON_BIN}" \
|
||||
RESULTS_ROOT="${RESULTS_ROOT}" \
|
||||
DOWNLOAD_ROOT="${VIS_DOWNLOAD_ROOT}" \
|
||||
OUTPUT_ROOT="${VIS_OUTPUT_ROOT}" \
|
||||
OUTPUT_DIR_NAME="${VIS_OUTPUT_DIR_NAME}" \
|
||||
TRACKING_JSON_NAME="${VIS_TRACKING_JSON_NAME}" \
|
||||
VIS_MAX_FRAMES="${VIS_MAX_FRAMES}" \
|
||||
VIS_CLASS_ID="${VIS_CLASS_ID}" \
|
||||
VIS_TRACK_IDS="${VIS_TRACK_IDS}" \
|
||||
VIS_SHOW_TRAJECTORY="${VIS_SHOW_TRAJECTORY}" \
|
||||
bash "${ISSUE_VISUALIZE_SCRIPT}" "${visualize_args[@]}"
|
||||
}
|
||||
|
||||
run_temporal_wrapper() {
|
||||
local target_path="$1"
|
||||
local temporal_args=("${target_path}")
|
||||
|
||||
for issue_id in "${ISSUE_IDS[@]}"; do
|
||||
temporal_args+=(--issue-id "${issue_id}")
|
||||
done
|
||||
|
||||
if [[ -n "${TEMPORAL_CLASS_ID}" ]]; then
|
||||
temporal_args+=(--class-id "${TEMPORAL_CLASS_ID}")
|
||||
fi
|
||||
|
||||
if [[ -n "${TEMPORAL_FRAME_ID_START}" ]]; then
|
||||
temporal_args+=(--frame-id-start "${TEMPORAL_FRAME_ID_START}")
|
||||
fi
|
||||
|
||||
if [[ -n "${TEMPORAL_FRAME_ID_END}" ]]; then
|
||||
temporal_args+=(--frame-id-end "${TEMPORAL_FRAME_ID_END}")
|
||||
fi
|
||||
|
||||
if [[ -n "${TEMPORAL_TRACK_IDS}" ]]; then
|
||||
# shellcheck disable=SC2206
|
||||
local temporal_track_id_arr=(${TEMPORAL_TRACK_IDS})
|
||||
for track_id in "${temporal_track_id_arr[@]}"; do
|
||||
temporal_args+=(--track-id "${track_id}")
|
||||
done
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "######################################################################"
|
||||
echo "# Feishu issue-data temporal observation"
|
||||
echo "######################################################################"
|
||||
echo "Target path : ${target_path}"
|
||||
if [[ -n "${TEMPORAL_CLASS_ID}" ]]; then
|
||||
echo "Class ID : ${TEMPORAL_CLASS_ID}"
|
||||
fi
|
||||
if [[ -n "${TEMPORAL_TRACK_IDS}" ]]; then
|
||||
echo "Track IDs : ${TEMPORAL_TRACK_IDS}"
|
||||
fi
|
||||
if [[ -n "${TEMPORAL_FRAME_ID_START}" || -n "${TEMPORAL_FRAME_ID_END}" ]]; then
|
||||
echo "Frame ID Range : [${TEMPORAL_FRAME_ID_START:-"-inf"}, ${TEMPORAL_FRAME_ID_END:-"+inf"}]"
|
||||
fi
|
||||
echo "Heading Source : ${TEMPORAL_HEADING_SOURCE}"
|
||||
|
||||
PYTHON_BIN="${PYTHON_BIN}" \
|
||||
RESULTS_ROOT="${RESULTS_ROOT}" \
|
||||
OUTPUT_ROOT="${TEMPORAL_OUTPUT_ROOT}" \
|
||||
OUTPUT_DIR_NAME="${TEMPORAL_OUTPUT_DIR_NAME}" \
|
||||
TRACKING_JSON_NAME="${TEMPORAL_JSON_NAME}" \
|
||||
TEMPORAL_MIN_LENGTH="${TEMPORAL_MIN_LENGTH}" \
|
||||
TEMPORAL_CLASS_ID="${TEMPORAL_CLASS_ID}" \
|
||||
TEMPORAL_TRACK_IDS="${TEMPORAL_TRACK_IDS}" \
|
||||
TEMPORAL_FRAME_ID_START="${TEMPORAL_FRAME_ID_START}" \
|
||||
TEMPORAL_FRAME_ID_END="${TEMPORAL_FRAME_ID_END}" \
|
||||
TEMPORAL_PLOTS="${TEMPORAL_PLOTS}" \
|
||||
TEMPORAL_EXPORT_SERIES="${TEMPORAL_EXPORT_SERIES}" \
|
||||
TEMPORAL_FOCUS_TRACK_PLOTS="${TEMPORAL_FOCUS_TRACK_PLOTS}" \
|
||||
TEMPORAL_PREFER_EGO="${TEMPORAL_PREFER_EGO}" \
|
||||
TEMPORAL_X_AXIS="${TEMPORAL_X_AXIS}" \
|
||||
TEMPORAL_HEADING_SOURCE="${TEMPORAL_HEADING_SOURCE}" \
|
||||
bash "${ISSUE_TEMPORAL_SCRIPT}" "${temporal_args[@]}"
|
||||
}
|
||||
|
||||
if [[ "${#ISSUE_IDS[@]}" -eq 0 ]]; then
|
||||
run_tracking_wrapper "${TARGET_PATH}"
|
||||
workflow_failed=0
|
||||
if [[ "${ENABLE_VISUALIZE}" == "1" ]]; then
|
||||
if ! run_visualization_wrapper "${TARGET_PATH}"; then
|
||||
workflow_failed=1
|
||||
fi
|
||||
fi
|
||||
if [[ "${ENABLE_TEMPORAL_ANALYSIS}" == "1" ]]; then
|
||||
if ! run_temporal_wrapper "${TARGET_PATH}"; then
|
||||
workflow_failed=1
|
||||
fi
|
||||
fi
|
||||
if [[ "${ENABLE_CONVERT}" == "1" ]]; then
|
||||
if ! run_conversion_wrapper "${TARGET_PATH}"; then
|
||||
workflow_failed=1
|
||||
fi
|
||||
fi
|
||||
[[ "${workflow_failed}" -eq 0 ]]
|
||||
exit 0
|
||||
fi
|
||||
|
||||
total_issues=0
|
||||
success_issues=0
|
||||
failed_issues=0
|
||||
|
||||
for issue_index in "${!ISSUE_IDS[@]}"; do
|
||||
issue_id="${ISSUE_IDS[issue_index]}"
|
||||
issue_root="${TARGET_PATH}/issue_${issue_id}"
|
||||
((total_issues += 1))
|
||||
|
||||
echo ""
|
||||
printf '[TRACKING][%d/%d] issue_%s\n' \
|
||||
"$((issue_index + 1))" "${#ISSUE_IDS[@]}" "${issue_id}"
|
||||
|
||||
if [[ ! -d "${issue_root}" ]]; then
|
||||
echo "[FAIL] issue_${issue_id}: not found under ${TARGET_PATH}" >&2
|
||||
((failed_issues += 1))
|
||||
continue
|
||||
fi
|
||||
|
||||
if run_tracking_wrapper "${issue_root}"; then
|
||||
((success_issues += 1))
|
||||
else
|
||||
((failed_issues += 1))
|
||||
echo "[FAIL] issue_${issue_id}: tracking failed" >&2
|
||||
fi
|
||||
done
|
||||
|
||||
echo ""
|
||||
printf '[DONE] issues=%d success=%d failed=%d\n' \
|
||||
"${total_issues}" "${success_issues}" "${failed_issues}"
|
||||
|
||||
workflow_failed=0
|
||||
if [[ "${ENABLE_VISUALIZE}" == "1" ]]; then
|
||||
if [[ "${failed_issues}" -ne 0 ]]; then
|
||||
echo "Skipping visualization because some tracking jobs failed" >&2
|
||||
else
|
||||
if ! run_visualization_wrapper "${TARGET_PATH}"; then
|
||||
workflow_failed=1
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
|
||||
if [[ "${ENABLE_TEMPORAL_ANALYSIS}" == "1" ]]; then
|
||||
if [[ "${failed_issues}" -ne 0 ]]; then
|
||||
echo "Skipping temporal observation because some tracking jobs failed" >&2
|
||||
else
|
||||
if ! run_temporal_wrapper "${TARGET_PATH}"; then
|
||||
workflow_failed=1
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
|
||||
if [[ "${ENABLE_CONVERT}" == "1" ]]; then
|
||||
if [[ "${failed_issues}" -ne 0 ]]; then
|
||||
echo "Skipping protocol conversion because some tracking jobs failed" >&2
|
||||
else
|
||||
if ! run_conversion_wrapper "${TARGET_PATH}"; then
|
||||
workflow_failed=1
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
|
||||
[[ "${failed_issues}" -eq 0 && "${workflow_failed}" -eq 0 ]]
|
||||
260
tools/feishu_project/run_issue_data_visualize_tracking.sh
Executable file
260
tools/feishu_project/run_issue_data_visualize_tracking.sh
Executable file
@@ -0,0 +1,260 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
|
||||
PROJECT_ROOT=$(cd "${SCRIPT_DIR}/../.." && pwd)
|
||||
export PYTHONPATH="${PROJECT_ROOT}:${PYTHONPATH:-}"
|
||||
|
||||
RESULTS_ROOT=${RESULTS_ROOT:-/data1/dongying/Mono3d/G1Q3/feishu_project/inference_issue_data}
|
||||
DOWNLOAD_ROOT=${DOWNLOAD_ROOT:-/data1/dongying/Mono3d/G1Q3/feishu_project/downloaded_issue_data}
|
||||
OUTPUT_ROOT=${OUTPUT_ROOT:-}
|
||||
OUTPUT_DIR_NAME=${OUTPUT_DIR_NAME:-tracking_vis_raw}
|
||||
PYTHON_BIN=${PYTHON_BIN:-/deeplearning_team/ydong/dongying/miniconda/envs/dev/bin/python}
|
||||
VISUALIZE_SCRIPT=${VISUALIZE_SCRIPT:-"${PROJECT_ROOT}/tools/temporal_analysis/visualize_tracking_boxes.py"}
|
||||
TRACKING_JSON_NAME=${TRACKING_JSON_NAME:-merge.json}
|
||||
VIS_MAX_FRAMES=${VIS_MAX_FRAMES:-}
|
||||
VIS_CLASS_ID=${VIS_CLASS_ID:-}
|
||||
VIS_TRACK_IDS=${VIS_TRACK_IDS:-}
|
||||
VIS_SHOW_TRAJECTORY=${VIS_SHOW_TRAJECTORY:-0}
|
||||
|
||||
TARGET_PATH=""
|
||||
ISSUE_IDS=()
|
||||
|
||||
while (($# > 0)); do
|
||||
case "$1" in
|
||||
--issue-id)
|
||||
if (($# < 2)); then
|
||||
echo "Error: --issue-id requires a value" >&2
|
||||
exit 1
|
||||
fi
|
||||
ISSUE_IDS+=("$2")
|
||||
shift 2
|
||||
;;
|
||||
--issue-id=*)
|
||||
ISSUE_IDS+=("${1#*=}")
|
||||
shift
|
||||
;;
|
||||
-*)
|
||||
echo "Error: unsupported option: $1" >&2
|
||||
exit 1
|
||||
;;
|
||||
*)
|
||||
if [[ -n "${TARGET_PATH}" ]]; then
|
||||
echo "Error: multiple target paths provided: ${TARGET_PATH} and $1" >&2
|
||||
exit 1
|
||||
fi
|
||||
TARGET_PATH="$1"
|
||||
shift
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
if [[ -z "${TARGET_PATH}" ]]; then
|
||||
TARGET_PATH="${RESULTS_ROOT}"
|
||||
fi
|
||||
|
||||
if [[ ! -e "${TARGET_PATH}" ]]; then
|
||||
echo "Error: target path does not exist: ${TARGET_PATH}" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
resolve_abs_path() {
|
||||
local target_path="$1"
|
||||
if [[ -d "${target_path}" ]]; then
|
||||
(
|
||||
cd "${target_path}"
|
||||
pwd -P
|
||||
)
|
||||
return 0
|
||||
fi
|
||||
|
||||
local parent_dir
|
||||
parent_dir=$(
|
||||
cd "$(dirname "${target_path}")"
|
||||
pwd -P
|
||||
)
|
||||
printf '%s/%s\n' "${parent_dir}" "$(basename "${target_path}")"
|
||||
}
|
||||
|
||||
RESULTS_ROOT_ABS="$(resolve_abs_path "${RESULTS_ROOT}")"
|
||||
DOWNLOAD_ROOT_ABS="$(resolve_abs_path "${DOWNLOAD_ROOT}")"
|
||||
|
||||
is_case_dir() {
|
||||
local dir_path="$1"
|
||||
[[ -d "${dir_path}" ]] && [[ -f "${dir_path}/${TRACKING_JSON_NAME}" ]]
|
||||
}
|
||||
|
||||
derive_output_dir() {
|
||||
local case_dir="$1"
|
||||
local case_abs
|
||||
local rel_case_dir
|
||||
|
||||
case_abs="$(resolve_abs_path "${case_dir}")"
|
||||
if [[ -n "${OUTPUT_ROOT}" ]]; then
|
||||
if [[ "${case_abs}" == "${RESULTS_ROOT_ABS}" ]]; then
|
||||
printf '%s\n' "${OUTPUT_ROOT}"
|
||||
return 0
|
||||
fi
|
||||
if [[ "${case_abs}" == "${RESULTS_ROOT_ABS}"/* ]]; then
|
||||
rel_case_dir="${case_abs#"${RESULTS_ROOT_ABS}/"}"
|
||||
printf '%s/%s\n' "${OUTPUT_ROOT%/}" "${rel_case_dir}"
|
||||
return 0
|
||||
fi
|
||||
printf '%s/%s\n' "${OUTPUT_ROOT%/}" "$(basename "${case_abs}")"
|
||||
return 0
|
||||
fi
|
||||
|
||||
printf '%s/%s\n' "${case_abs}" "${OUTPUT_DIR_NAME}"
|
||||
}
|
||||
|
||||
derive_raw_video_path() {
|
||||
local case_dir="$1"
|
||||
local case_abs
|
||||
local rel_case_dir
|
||||
|
||||
case_abs="$(resolve_abs_path "${case_dir}")"
|
||||
if [[ "${case_abs}" != "${RESULTS_ROOT_ABS}" && "${case_abs}" != "${RESULTS_ROOT_ABS}"/* ]]; then
|
||||
echo "Error: case directory is not under RESULTS_ROOT: ${case_abs}" >&2
|
||||
return 1
|
||||
fi
|
||||
|
||||
if [[ "${case_abs}" == "${RESULTS_ROOT_ABS}" ]]; then
|
||||
echo "Error: cannot derive a raw video path from the results root itself" >&2
|
||||
return 1
|
||||
fi
|
||||
|
||||
rel_case_dir="${case_abs#"${RESULTS_ROOT_ABS}/"}"
|
||||
printf '%s/%s/sigmastar.1/camera4.bin\n' "${DOWNLOAD_ROOT_ABS}" "${rel_case_dir}"
|
||||
}
|
||||
|
||||
run_single_case() {
|
||||
local case_dir="$1"
|
||||
local tracking_json="${case_dir}/${TRACKING_JSON_NAME}"
|
||||
local raw_video_path
|
||||
local output_dir
|
||||
local cmd
|
||||
|
||||
if [[ ! -f "${tracking_json}" ]]; then
|
||||
echo "[ERROR] ${TRACKING_JSON_NAME} not found in case directory: ${case_dir}" >&2
|
||||
return 1
|
||||
fi
|
||||
|
||||
raw_video_path="$(derive_raw_video_path "${case_dir}")"
|
||||
if [[ ! -f "${raw_video_path}" ]]; then
|
||||
echo "[ERROR] camera4.bin not found for case: ${case_dir}" >&2
|
||||
echo " expected: ${raw_video_path}" >&2
|
||||
return 1
|
||||
fi
|
||||
|
||||
output_dir="$(derive_output_dir "${case_dir}")"
|
||||
|
||||
cmd=(
|
||||
"${PYTHON_BIN}" "${VISUALIZE_SCRIPT}"
|
||||
--tracking "${tracking_json}"
|
||||
--images "${raw_video_path}"
|
||||
--output "${output_dir}"
|
||||
--align-by-frame-id
|
||||
)
|
||||
|
||||
if [[ -n "${VIS_MAX_FRAMES}" ]]; then
|
||||
cmd+=(--max-frames "${VIS_MAX_FRAMES}")
|
||||
fi
|
||||
|
||||
if [[ -n "${VIS_CLASS_ID}" ]]; then
|
||||
cmd+=(--class-id "${VIS_CLASS_ID}")
|
||||
fi
|
||||
|
||||
if [[ -n "${VIS_TRACK_IDS}" ]]; then
|
||||
# shellcheck disable=SC2206
|
||||
local track_id_arr=(${VIS_TRACK_IDS})
|
||||
cmd+=(--track-ids "${track_id_arr[@]}")
|
||||
fi
|
||||
|
||||
if [[ "${VIS_SHOW_TRAJECTORY}" == "1" ]]; then
|
||||
cmd+=(--show-trajectory)
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "######################################################################"
|
||||
echo "# Feishu issue-data raw-image visualization"
|
||||
echo "######################################################################"
|
||||
echo "Case : ${case_dir}"
|
||||
echo "Track : ${tracking_json}"
|
||||
echo "Video : ${raw_video_path}"
|
||||
echo "Output : ${output_dir}"
|
||||
|
||||
"${cmd[@]}"
|
||||
}
|
||||
|
||||
run_batch_root() {
|
||||
local batch_root="$1"
|
||||
local total_cases=0
|
||||
local success_cases=0
|
||||
local failed_cases=0
|
||||
|
||||
while IFS= read -r -d '' tracking_json; do
|
||||
local case_dir
|
||||
case_dir="$(dirname "${tracking_json}")"
|
||||
((total_cases += 1))
|
||||
|
||||
if run_single_case "${case_dir}"; then
|
||||
((success_cases += 1))
|
||||
else
|
||||
((failed_cases += 1))
|
||||
printf '[FAIL] case=%s\n' "${case_dir}" >&2
|
||||
fi
|
||||
done < <(
|
||||
find "${batch_root}" -type f -name "${TRACKING_JSON_NAME}" -print0 | sort -z
|
||||
)
|
||||
|
||||
if [[ "${total_cases}" -eq 0 ]]; then
|
||||
echo "[ERROR] No ${TRACKING_JSON_NAME} files were found under: ${batch_root}" >&2
|
||||
return 1
|
||||
fi
|
||||
|
||||
echo ""
|
||||
printf '[DONE] cases=%d success=%d failed=%d\n' \
|
||||
"${total_cases}" "${success_cases}" "${failed_cases}"
|
||||
|
||||
[[ "${failed_cases}" -eq 0 ]]
|
||||
}
|
||||
|
||||
if [[ "${#ISSUE_IDS[@]}" -eq 0 ]]; then
|
||||
if is_case_dir "${TARGET_PATH}"; then
|
||||
run_single_case "${TARGET_PATH}"
|
||||
elif [[ -d "${TARGET_PATH}" ]]; then
|
||||
run_batch_root "${TARGET_PATH}"
|
||||
else
|
||||
echo "Error: unsupported target path: ${TARGET_PATH}" >&2
|
||||
exit 1
|
||||
fi
|
||||
exit 0
|
||||
fi
|
||||
|
||||
total_issues=0
|
||||
success_issues=0
|
||||
failed_issues=0
|
||||
|
||||
for issue_id in "${ISSUE_IDS[@]}"; do
|
||||
issue_root="${TARGET_PATH}/issue_${issue_id}"
|
||||
((total_issues += 1))
|
||||
|
||||
if [[ ! -d "${issue_root}" ]]; then
|
||||
echo "[FAIL] issue_${issue_id}: not found under ${TARGET_PATH}" >&2
|
||||
((failed_issues += 1))
|
||||
continue
|
||||
fi
|
||||
|
||||
if run_batch_root "${issue_root}"; then
|
||||
((success_issues += 1))
|
||||
else
|
||||
((failed_issues += 1))
|
||||
echo "[FAIL] issue_${issue_id}: visualization failed" >&2
|
||||
fi
|
||||
done
|
||||
|
||||
echo ""
|
||||
printf '[DONE] issues=%d success=%d failed=%d\n' \
|
||||
"${total_issues}" "${success_issues}" "${failed_issues}"
|
||||
|
||||
[[ "${failed_issues}" -eq 0 ]]
|
||||
23
tools/feishu_project/run_issue_tag_profile.sh
Executable file
23
tools/feishu_project/run_issue_tag_profile.sh
Executable file
@@ -0,0 +1,23 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
|
||||
PROJECT_ROOT=$(cd "${SCRIPT_DIR}/../.." && pwd)
|
||||
|
||||
SYNC_ROOT=${SYNC_ROOT:-/data1/dongying/Mono3d/G1Q3/feishu_project}
|
||||
INPUT_JSON=${INPUT_JSON:-"${SYNC_ROOT}/exports/dongying_g1q3_issue_list.json"}
|
||||
OUTPUT_DIR=${OUTPUT_DIR:-"${SYNC_ROOT}/reports/issue_tag_profile"}
|
||||
REPORT_NAME=${REPORT_NAME:-dongying_g1q3_issue_tag_profile}
|
||||
PYTHON_BIN=${PYTHON_BIN:-/deeplearning_team/ydong/dongying/miniconda/envs/dev/bin/python}
|
||||
TOP_LIMIT=${TOP_LIMIT:-15}
|
||||
|
||||
CMD=(
|
||||
"${PYTHON_BIN}" "${PROJECT_ROOT}/tools/feishu_project/analyze_issue_tag_profile.py"
|
||||
--input-json "${INPUT_JSON}"
|
||||
--output-dir "${OUTPUT_DIR}"
|
||||
--report-name "${REPORT_NAME}"
|
||||
--top-limit "${TOP_LIMIT}"
|
||||
)
|
||||
|
||||
CMD+=("$@")
|
||||
"${CMD[@]}"
|
||||
583
tools/feishu_project/run_roi1_crop_compensation_experiment.py
Executable file
583
tools/feishu_project/run_roi1_crop_compensation_experiment.py
Executable file
@@ -0,0 +1,583 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Run ROI1 crop-center compensation experiments and compare detection stability."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import math
|
||||
import re
|
||||
import statistics
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
DEFAULT_MODEL = ROOT / "runs" / "export" / "train_mono3d_two_roi_20260416-raw_no_edge" / "merged_model.torchscript"
|
||||
DEFAULT_OUTPUT_ROOT = Path("/data1/dongying/Mono3d/G1Q3/feishu_project/roi1_crop_compensation_experiments")
|
||||
FRAME_FILE_RE = re.compile(r"camera4_(\d+)_")
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--baseline-case-dir", required=True, type=Path, help="Baseline inference case output directory.")
|
||||
parser.add_argument("--video-case-dir", required=True, type=Path, help="Input video case path passed to --video-case-dir.")
|
||||
parser.add_argument("--tracking-json", type=Path, default=None, help="Tracking JSON used to derive the reference target trajectory.")
|
||||
parser.add_argument("--track-id", type=int, default=7)
|
||||
parser.add_argument("--frame-id-start", type=int, required=True)
|
||||
parser.add_argument("--frame-id-end", type=int, required=True)
|
||||
parser.add_argument("--alpha", type=float, default=1.0, help="Scale factor applied to the bbox-derived ROI1 compensation.")
|
||||
parser.add_argument("--exported-model", type=Path, default=DEFAULT_MODEL)
|
||||
parser.add_argument("--device", type=str, default="cuda")
|
||||
parser.add_argument("--output-root", type=Path, default=DEFAULT_OUTPUT_ROOT)
|
||||
parser.add_argument("--skip-existing", action="store_true", help="Reuse existing experiment outputs when present.")
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def load_json(path: Path) -> Any:
|
||||
with path.open("r", encoding="utf-8") as file:
|
||||
return json.load(file)
|
||||
|
||||
|
||||
def save_json(path: Path, payload: Any) -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with path.open("w", encoding="utf-8") as file:
|
||||
json.dump(payload, file, ensure_ascii=False, indent=2)
|
||||
file.write("\n")
|
||||
|
||||
|
||||
def coerce_bbox(values: Any) -> tuple[float, float, float, float] | None:
|
||||
if not isinstance(values, (list, tuple)) or len(values) < 4:
|
||||
return None
|
||||
try:
|
||||
x1, y1, x2, y2 = (float(values[0]), float(values[1]), float(values[2]), float(values[3]))
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
return x1, y1, x2, y2
|
||||
|
||||
|
||||
def bbox_iou(box_a: tuple[float, float, float, float], box_b: tuple[float, float, float, float]) -> float:
|
||||
ax1, ay1, ax2, ay2 = box_a
|
||||
bx1, by1, bx2, by2 = box_b
|
||||
inter_x1 = max(ax1, bx1)
|
||||
inter_y1 = max(ay1, by1)
|
||||
inter_x2 = min(ax2, bx2)
|
||||
inter_y2 = min(ay2, by2)
|
||||
inter_w = max(0.0, inter_x2 - inter_x1)
|
||||
inter_h = max(0.0, inter_y2 - inter_y1)
|
||||
inter_area = inter_w * inter_h
|
||||
if inter_area <= 0.0:
|
||||
return 0.0
|
||||
area_a = max(0.0, ax2 - ax1) * max(0.0, ay2 - ay1)
|
||||
area_b = max(0.0, bx2 - bx1) * max(0.0, by2 - by1)
|
||||
denom = area_a + area_b - inter_area
|
||||
return inter_area / denom if denom > 0.0 else 0.0
|
||||
|
||||
|
||||
def center_distance(box_a: tuple[float, float, float, float], box_b: tuple[float, float, float, float]) -> float:
|
||||
acx = (box_a[0] + box_a[2]) * 0.5
|
||||
acy = (box_a[1] + box_a[3]) * 0.5
|
||||
bcx = (box_b[0] + box_b[2]) * 0.5
|
||||
bcy = (box_b[1] + box_b[3]) * 0.5
|
||||
return math.hypot(acx - bcx, acy - bcy)
|
||||
|
||||
|
||||
def frame_id_from_tracking_frame(frame_data: dict[str, Any], fallback_idx: int) -> int:
|
||||
for det in frame_data.get("detections", []):
|
||||
for key in ("frameId", "frame_id"):
|
||||
value = det.get(key)
|
||||
if value is None:
|
||||
continue
|
||||
try:
|
||||
return int(value)
|
||||
except (TypeError, ValueError):
|
||||
continue
|
||||
frame_info = frame_data.get("frame_info")
|
||||
if isinstance(frame_info, dict):
|
||||
for key in ("frame_id", "frameId", "original_frame_id"):
|
||||
value = frame_info.get(key)
|
||||
if value is None:
|
||||
continue
|
||||
try:
|
||||
return int(value)
|
||||
except (TypeError, ValueError):
|
||||
continue
|
||||
return fallback_idx
|
||||
|
||||
|
||||
def load_reference_track(tracking_json: Path, track_id: int, frame_id_start: int, frame_id_end: int) -> list[dict[str, Any]]:
|
||||
payload = load_json(tracking_json)
|
||||
frames = payload.get("frames", payload) if isinstance(payload, dict) else payload
|
||||
if not isinstance(frames, list):
|
||||
raise ValueError(f"Unsupported tracking JSON structure in {tracking_json}")
|
||||
|
||||
reference = []
|
||||
for frame_idx, frame_data in enumerate(frames):
|
||||
if not isinstance(frame_data, dict):
|
||||
continue
|
||||
frame_id = frame_id_from_tracking_frame(frame_data, frame_idx)
|
||||
if frame_id < frame_id_start or frame_id > frame_id_end:
|
||||
continue
|
||||
for det in frame_data.get("detections", []):
|
||||
if det.get("track_id") != track_id:
|
||||
continue
|
||||
bbox = coerce_bbox(det.get("bbox"))
|
||||
if bbox is None:
|
||||
continue
|
||||
class_id = int(det.get("class_id")) if det.get("class_id") is not None else None
|
||||
reference.append(
|
||||
{
|
||||
"frame_id": frame_id,
|
||||
"frame_idx": frame_idx,
|
||||
"bbox": bbox,
|
||||
"class_id": class_id,
|
||||
"type_name": det.get("type_name"),
|
||||
"y2": bbox[3],
|
||||
}
|
||||
)
|
||||
break
|
||||
|
||||
reference.sort(key=lambda item: item["frame_id"])
|
||||
if not reference:
|
||||
raise FileNotFoundError(
|
||||
f"Track {track_id} not found in {tracking_json} within frame_id [{frame_id_start}, {frame_id_end}]"
|
||||
)
|
||||
return reference
|
||||
|
||||
|
||||
def build_offset_maps(
|
||||
reference_track: list[dict[str, Any]],
|
||||
alpha: float,
|
||||
) -> tuple[dict[int, float], dict[int, float], dict[int, float]]:
|
||||
ref_y2 = float(reference_track[0]["y2"])
|
||||
oracle: dict[int, float] = {}
|
||||
causal: dict[int, float] = {}
|
||||
frame_delta: dict[int, float] = {}
|
||||
|
||||
prev_oracle_offset = 0.0
|
||||
prev_y2 = ref_y2
|
||||
for idx, item in enumerate(reference_track):
|
||||
frame_id = int(item["frame_id"])
|
||||
current_offset = alpha * (float(item["y2"]) - ref_y2)
|
||||
oracle[frame_id] = current_offset
|
||||
causal[frame_id] = prev_oracle_offset if idx > 0 else 0.0
|
||||
frame_delta[frame_id] = alpha * (float(item["y2"]) - prev_y2) if idx > 0 else 0.0
|
||||
prev_oracle_offset = current_offset
|
||||
prev_y2 = float(item["y2"])
|
||||
return oracle, causal, frame_delta
|
||||
|
||||
|
||||
def write_offset_map(path: Path, offsets: dict[int, float], metadata: dict[str, Any]) -> None:
|
||||
save_json(
|
||||
path,
|
||||
{
|
||||
"default_offset_px": 0.0,
|
||||
"frame_id_offsets": {str(frame_id): offset for frame_id, offset in offsets.items()},
|
||||
"metadata": metadata,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def build_prediction_frame_map(predictions_merge_dir: Path) -> dict[int, Path]:
|
||||
frame_map: dict[int, Path] = {}
|
||||
for json_path in sorted(predictions_merge_dir.glob("*.json")):
|
||||
match = FRAME_FILE_RE.search(json_path.name)
|
||||
if not match:
|
||||
continue
|
||||
frame_map[int(match.group(1))] = json_path
|
||||
return frame_map
|
||||
|
||||
|
||||
def extract_candidate_records(frame_json_path: Path) -> list[dict[str, Any]]:
|
||||
payload = load_json(frame_json_path)
|
||||
if not isinstance(payload, dict):
|
||||
raise ValueError(f"Unexpected frame prediction structure in {frame_json_path}")
|
||||
|
||||
candidates = []
|
||||
for value in payload.values():
|
||||
if not isinstance(value, dict):
|
||||
continue
|
||||
bbox = coerce_bbox(value.get("box2d"))
|
||||
if bbox is None:
|
||||
continue
|
||||
class_id = value.get("type")
|
||||
try:
|
||||
class_id = int(class_id) if class_id is not None else None
|
||||
except (TypeError, ValueError):
|
||||
class_id = None
|
||||
x_ego = None
|
||||
if isinstance(value.get("box_center_xyz_ego"), list) and value["box_center_xyz_ego"]:
|
||||
try:
|
||||
x_ego = float(value["box_center_xyz_ego"][0])
|
||||
except (TypeError, ValueError):
|
||||
x_ego = None
|
||||
if x_ego is None and isinstance(value.get("xyzlhwyaw_ego"), list) and value["xyzlhwyaw_ego"]:
|
||||
try:
|
||||
x_ego = float(value["xyzlhwyaw_ego"][0])
|
||||
except (TypeError, ValueError):
|
||||
x_ego = None
|
||||
if x_ego is None:
|
||||
continue
|
||||
candidates.append(
|
||||
{
|
||||
"bbox": bbox,
|
||||
"class_id": class_id,
|
||||
"type_name": value.get("type_name"),
|
||||
"score": float(value.get("score", 0.0)),
|
||||
"x_ego": x_ego,
|
||||
}
|
||||
)
|
||||
return candidates
|
||||
|
||||
|
||||
def match_reference_detection(
|
||||
frame_json_path: Path,
|
||||
reference_bbox: tuple[float, float, float, float],
|
||||
reference_class_id: int | None,
|
||||
) -> dict[str, Any] | None:
|
||||
candidates = extract_candidate_records(frame_json_path)
|
||||
same_class = [
|
||||
candidate for candidate in candidates if reference_class_id is not None and candidate["class_id"] == reference_class_id
|
||||
]
|
||||
candidate_pool = same_class or candidates
|
||||
if not candidate_pool:
|
||||
return None
|
||||
|
||||
best = max(
|
||||
candidate_pool,
|
||||
key=lambda candidate: (
|
||||
bbox_iou(candidate["bbox"], reference_bbox),
|
||||
-center_distance(candidate["bbox"], reference_bbox),
|
||||
candidate["score"],
|
||||
),
|
||||
)
|
||||
best = dict(best)
|
||||
best["iou_to_reference"] = bbox_iou(best["bbox"], reference_bbox)
|
||||
return best
|
||||
|
||||
|
||||
def collect_variant_series(reference_track: list[dict[str, Any]], predictions_merge_dir: Path) -> list[dict[str, Any]]:
|
||||
frame_map = build_prediction_frame_map(predictions_merge_dir)
|
||||
series: list[dict[str, Any]] = []
|
||||
for item in reference_track:
|
||||
frame_id = int(item["frame_id"])
|
||||
frame_json_path = frame_map.get(frame_id)
|
||||
if frame_json_path is None:
|
||||
continue
|
||||
matched = match_reference_detection(frame_json_path, item["bbox"], item["class_id"])
|
||||
if matched is None:
|
||||
continue
|
||||
bbox = matched["bbox"]
|
||||
series.append(
|
||||
{
|
||||
"frame_id": frame_id,
|
||||
"x_ego": float(matched["x_ego"]),
|
||||
"score": float(matched["score"]),
|
||||
"iou_to_reference": float(matched["iou_to_reference"]),
|
||||
"bbox": bbox,
|
||||
"y2": bbox[3],
|
||||
"cy": (bbox[1] + bbox[3]) * 0.5,
|
||||
"w": bbox[2] - bbox[0],
|
||||
"h": bbox[3] - bbox[1],
|
||||
}
|
||||
)
|
||||
series.sort(key=lambda item: item["frame_id"])
|
||||
return series
|
||||
|
||||
|
||||
def percentile(sorted_values: list[float], q: float) -> float:
|
||||
if not sorted_values:
|
||||
raise ValueError("sorted_values must not be empty")
|
||||
if q <= 0:
|
||||
return sorted_values[0]
|
||||
if q >= 1:
|
||||
return sorted_values[-1]
|
||||
index = max(0, math.ceil(q * len(sorted_values)) - 1)
|
||||
return sorted_values[index]
|
||||
|
||||
|
||||
def compute_series_metrics(series: list[dict[str, Any]]) -> dict[str, Any]:
|
||||
if len(series) < 2:
|
||||
raise ValueError("Series must contain at least two samples")
|
||||
|
||||
x_values = [item["x_ego"] for item in series]
|
||||
y2_values = [item["y2"] for item in series]
|
||||
cy_values = [item["cy"] for item in series]
|
||||
ious = [item["iou_to_reference"] for item in series]
|
||||
scores = [item["score"] for item in series]
|
||||
|
||||
dx = [x_values[idx] - x_values[idx - 1] for idx in range(1, len(x_values))]
|
||||
dy2 = [y2_values[idx] - y2_values[idx - 1] for idx in range(1, len(y2_values))]
|
||||
dcy = [cy_values[idx] - cy_values[idx - 1] for idx in range(1, len(cy_values))]
|
||||
abs_dx = sorted(abs(value) for value in dx)
|
||||
abs_dy2 = sorted(abs(value) for value in dy2)
|
||||
abs_dcy = sorted(abs(value) for value in dcy)
|
||||
|
||||
window = 5
|
||||
local_dev = []
|
||||
for idx in range(window, len(x_values) - window):
|
||||
local_mean = sum(x_values[idx - window : idx + window + 1]) / (2 * window + 1)
|
||||
local_dev.append(abs(x_values[idx] - local_mean))
|
||||
local_dev_sorted = sorted(local_dev) if local_dev else [0.0]
|
||||
|
||||
return {
|
||||
"samples": len(series),
|
||||
"frame_id_start": int(series[0]["frame_id"]),
|
||||
"frame_id_end": int(series[-1]["frame_id"]),
|
||||
"x_start": float(x_values[0]),
|
||||
"x_end": float(x_values[-1]),
|
||||
"x_change": float(x_values[-1] - x_values[0]),
|
||||
"abs_dx_mean": float(statistics.mean(abs(value) for value in dx)),
|
||||
"abs_dx_p95": float(percentile(abs_dx, 0.95)),
|
||||
"abs_dx_max": float(max(abs_dx)),
|
||||
"abs_dy2_mean": float(statistics.mean(abs(value) for value in dy2)),
|
||||
"abs_dy2_p95": float(percentile(abs_dy2, 0.95)),
|
||||
"abs_dcy_mean": float(statistics.mean(abs(value) for value in dcy)),
|
||||
"abs_dcy_p95": float(percentile(abs_dcy, 0.95)),
|
||||
"local_dev_mean": float(statistics.mean(local_dev_sorted)),
|
||||
"local_dev_p95": float(percentile(local_dev_sorted, 0.95)),
|
||||
"local_dev_max": float(max(local_dev_sorted)),
|
||||
"mean_iou_to_reference": float(statistics.mean(ious)),
|
||||
"min_iou_to_reference": float(min(ious)),
|
||||
"mean_score": float(statistics.mean(scores)),
|
||||
}
|
||||
|
||||
|
||||
def run_inference_variant(
|
||||
*,
|
||||
video_case_dir: Path,
|
||||
output_dir: Path,
|
||||
frame_id_start: int,
|
||||
frame_id_end: int,
|
||||
exported_model: Path,
|
||||
device: str,
|
||||
offset_map_path: Path | None = None,
|
||||
) -> None:
|
||||
cmd = [
|
||||
sys.executable,
|
||||
str(ROOT / "tools" / "model_inference" / "core" / "run_two_roi_exported_onnx_infer.py"),
|
||||
"--video-case-dir",
|
||||
str(video_case_dir),
|
||||
"--output-dir",
|
||||
str(output_dir),
|
||||
"--frame-id-start",
|
||||
str(frame_id_start),
|
||||
"--frame-id-end",
|
||||
str(frame_id_end),
|
||||
"--video-stride",
|
||||
"1",
|
||||
"--exported-model",
|
||||
str(exported_model),
|
||||
"--device",
|
||||
device,
|
||||
"--enable-cross-class-merge-prior",
|
||||
"--save-aggregate-predictions",
|
||||
]
|
||||
if offset_map_path is not None:
|
||||
cmd.extend(["--roi1-crop-center-y-offset-map", str(offset_map_path)])
|
||||
subprocess.run(cmd, check=True, cwd=str(ROOT))
|
||||
|
||||
|
||||
def build_report_payload(
|
||||
*,
|
||||
args: argparse.Namespace,
|
||||
reference_track: list[dict[str, Any]],
|
||||
baseline_metrics: dict[str, Any],
|
||||
oracle_metrics: dict[str, Any],
|
||||
causal_metrics: dict[str, Any],
|
||||
frame_delta_metrics: dict[str, Any],
|
||||
oracle_offsets_path: Path,
|
||||
causal_offsets_path: Path,
|
||||
frame_delta_offsets_path: Path,
|
||||
baseline_dir: Path,
|
||||
oracle_dir: Path,
|
||||
causal_dir: Path,
|
||||
frame_delta_dir: Path,
|
||||
) -> dict[str, Any]:
|
||||
baseline_abs_dx_p95 = baseline_metrics["abs_dx_p95"]
|
||||
baseline_local_dev_p95 = baseline_metrics["local_dev_p95"]
|
||||
|
||||
def compare_metrics(metrics: dict[str, Any]) -> dict[str, Any]:
|
||||
return {
|
||||
"abs_dx_p95_delta": float(metrics["abs_dx_p95"] - baseline_abs_dx_p95),
|
||||
"abs_dx_p95_reduction_ratio": float((baseline_abs_dx_p95 - metrics["abs_dx_p95"]) / baseline_abs_dx_p95)
|
||||
if baseline_abs_dx_p95 > 0
|
||||
else 0.0,
|
||||
"local_dev_p95_delta": float(metrics["local_dev_p95"] - baseline_local_dev_p95),
|
||||
"local_dev_p95_reduction_ratio": float((baseline_local_dev_p95 - metrics["local_dev_p95"]) / baseline_local_dev_p95)
|
||||
if baseline_local_dev_p95 > 0
|
||||
else 0.0,
|
||||
}
|
||||
|
||||
return {
|
||||
"track_id": int(args.track_id),
|
||||
"frame_id_start": int(args.frame_id_start),
|
||||
"frame_id_end": int(args.frame_id_end),
|
||||
"alpha": float(args.alpha),
|
||||
"exported_model": str(args.exported_model.resolve()),
|
||||
"baseline_case_dir": str(baseline_dir.resolve()),
|
||||
"video_case_dir": str(args.video_case_dir.resolve()),
|
||||
"reference_track_frames": len(reference_track),
|
||||
"oracle_offset_map": str(oracle_offsets_path.resolve()),
|
||||
"causal_offset_map": str(causal_offsets_path.resolve()),
|
||||
"frame_delta_offset_map": str(frame_delta_offsets_path.resolve()),
|
||||
"variants": {
|
||||
"baseline": {
|
||||
"output_dir": str(baseline_dir.resolve()),
|
||||
"metrics": baseline_metrics,
|
||||
},
|
||||
"oracle": {
|
||||
"output_dir": str(oracle_dir.resolve()),
|
||||
"metrics": oracle_metrics,
|
||||
"comparison_to_baseline": compare_metrics(oracle_metrics),
|
||||
},
|
||||
"causal": {
|
||||
"output_dir": str(causal_dir.resolve()),
|
||||
"metrics": causal_metrics,
|
||||
"comparison_to_baseline": compare_metrics(causal_metrics),
|
||||
},
|
||||
"frame_delta": {
|
||||
"output_dir": str(frame_delta_dir.resolve()),
|
||||
"metrics": frame_delta_metrics,
|
||||
"comparison_to_baseline": compare_metrics(frame_delta_metrics),
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def main() -> int:
|
||||
args = parse_args()
|
||||
baseline_case_dir = args.baseline_case_dir.resolve()
|
||||
tracking_json = args.tracking_json.resolve() if args.tracking_json else baseline_case_dir / "merge.json"
|
||||
|
||||
reference_track = load_reference_track(
|
||||
tracking_json=tracking_json,
|
||||
track_id=args.track_id,
|
||||
frame_id_start=args.frame_id_start,
|
||||
frame_id_end=args.frame_id_end,
|
||||
)
|
||||
oracle_offsets, causal_offsets, frame_delta_offsets = build_offset_maps(reference_track, alpha=args.alpha)
|
||||
|
||||
case_tag = f"{baseline_case_dir.name}_track{args.track_id}_f{args.frame_id_start}_{args.frame_id_end}_a{str(args.alpha).replace('.', 'p')}"
|
||||
output_root = args.output_root.resolve() / case_tag
|
||||
configs_dir = output_root / "configs"
|
||||
oracle_offsets_path = configs_dir / "oracle_roi1_offsets.json"
|
||||
causal_offsets_path = configs_dir / "causal_prev_frame_roi1_offsets.json"
|
||||
frame_delta_offsets_path = configs_dir / "frame_delta_roi1_offsets.json"
|
||||
oracle_dir = output_root / "oracle"
|
||||
causal_dir = output_root / "causal"
|
||||
frame_delta_dir = output_root / "frame_delta"
|
||||
report_path = output_root / "experiment_summary.json"
|
||||
|
||||
write_offset_map(
|
||||
oracle_offsets_path,
|
||||
oracle_offsets,
|
||||
{
|
||||
"mode": "same_frame_oracle",
|
||||
"track_id": args.track_id,
|
||||
"alpha": args.alpha,
|
||||
"reference_frame_id": reference_track[0]["frame_id"],
|
||||
"reference_y2": reference_track[0]["y2"],
|
||||
},
|
||||
)
|
||||
write_offset_map(
|
||||
causal_offsets_path,
|
||||
causal_offsets,
|
||||
{
|
||||
"mode": "previous_frame_causal",
|
||||
"track_id": args.track_id,
|
||||
"alpha": args.alpha,
|
||||
"reference_frame_id": reference_track[0]["frame_id"],
|
||||
"reference_y2": reference_track[0]["y2"],
|
||||
},
|
||||
)
|
||||
write_offset_map(
|
||||
frame_delta_offsets_path,
|
||||
frame_delta_offsets,
|
||||
{
|
||||
"mode": "same_frame_prev_delta",
|
||||
"track_id": args.track_id,
|
||||
"alpha": args.alpha,
|
||||
"reference_frame_id": reference_track[0]["frame_id"],
|
||||
"reference_y2": reference_track[0]["y2"],
|
||||
},
|
||||
)
|
||||
|
||||
if not args.skip_existing or not (oracle_dir / "predictions" / "merge").is_dir():
|
||||
run_inference_variant(
|
||||
video_case_dir=args.video_case_dir.resolve(),
|
||||
output_dir=oracle_dir,
|
||||
frame_id_start=args.frame_id_start,
|
||||
frame_id_end=args.frame_id_end,
|
||||
exported_model=args.exported_model.resolve(),
|
||||
device=args.device,
|
||||
offset_map_path=oracle_offsets_path,
|
||||
)
|
||||
|
||||
if not args.skip_existing or not (causal_dir / "predictions" / "merge").is_dir():
|
||||
run_inference_variant(
|
||||
video_case_dir=args.video_case_dir.resolve(),
|
||||
output_dir=causal_dir,
|
||||
frame_id_start=args.frame_id_start,
|
||||
frame_id_end=args.frame_id_end,
|
||||
exported_model=args.exported_model.resolve(),
|
||||
device=args.device,
|
||||
offset_map_path=causal_offsets_path,
|
||||
)
|
||||
|
||||
if not args.skip_existing or not (frame_delta_dir / "predictions" / "merge").is_dir():
|
||||
run_inference_variant(
|
||||
video_case_dir=args.video_case_dir.resolve(),
|
||||
output_dir=frame_delta_dir,
|
||||
frame_id_start=args.frame_id_start,
|
||||
frame_id_end=args.frame_id_end,
|
||||
exported_model=args.exported_model.resolve(),
|
||||
device=args.device,
|
||||
offset_map_path=frame_delta_offsets_path,
|
||||
)
|
||||
|
||||
baseline_series = collect_variant_series(reference_track, baseline_case_dir / "predictions" / "merge")
|
||||
oracle_series = collect_variant_series(reference_track, oracle_dir / "predictions" / "merge")
|
||||
causal_series = collect_variant_series(reference_track, causal_dir / "predictions" / "merge")
|
||||
frame_delta_series = collect_variant_series(reference_track, frame_delta_dir / "predictions" / "merge")
|
||||
|
||||
baseline_metrics = compute_series_metrics(baseline_series)
|
||||
oracle_metrics = compute_series_metrics(oracle_series)
|
||||
causal_metrics = compute_series_metrics(causal_series)
|
||||
frame_delta_metrics = compute_series_metrics(frame_delta_series)
|
||||
|
||||
report_payload = build_report_payload(
|
||||
args=args,
|
||||
reference_track=reference_track,
|
||||
baseline_metrics=baseline_metrics,
|
||||
oracle_metrics=oracle_metrics,
|
||||
causal_metrics=causal_metrics,
|
||||
frame_delta_metrics=frame_delta_metrics,
|
||||
oracle_offsets_path=oracle_offsets_path,
|
||||
causal_offsets_path=causal_offsets_path,
|
||||
frame_delta_offsets_path=frame_delta_offsets_path,
|
||||
baseline_dir=baseline_case_dir,
|
||||
oracle_dir=oracle_dir,
|
||||
causal_dir=causal_dir,
|
||||
frame_delta_dir=frame_delta_dir,
|
||||
)
|
||||
save_json(report_path, report_payload)
|
||||
|
||||
print("")
|
||||
print("ROI1 crop compensation experiment summary")
|
||||
print(f"summary_json: {report_path}")
|
||||
print(f"baseline abs_dx_p95 : {baseline_metrics['abs_dx_p95']:.4f} m/frame")
|
||||
print(f"oracle abs_dx_p95 : {oracle_metrics['abs_dx_p95']:.4f} m/frame")
|
||||
print(f"causal abs_dx_p95 : {causal_metrics['abs_dx_p95']:.4f} m/frame")
|
||||
print(f"delta abs_dx_p95 : {frame_delta_metrics['abs_dx_p95']:.4f} m/frame")
|
||||
print(f"baseline local_dev_p95 : {baseline_metrics['local_dev_p95']:.4f} m")
|
||||
print(f"oracle local_dev_p95 : {oracle_metrics['local_dev_p95']:.4f} m")
|
||||
print(f"causal local_dev_p95 : {causal_metrics['local_dev_p95']:.4f} m")
|
||||
print(f"delta local_dev_p95 : {frame_delta_metrics['local_dev_p95']:.4f} m")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
219
tools/feishu_project/skill/feishu-doc/SKILL.md
Executable file
219
tools/feishu_project/skill/feishu-doc/SKILL.md
Executable file
@@ -0,0 +1,219 @@
|
||||
---
|
||||
name: feishu-doc
|
||||
description: Read, create, append, replace, and structurally edit Feishu Docx documents with the feishu_doc tool. Use when Codex needs to work with Feishu docs, cloud docs, docx links, document blocks, tables, images, or file attachments in Feishu/Lark documents.
|
||||
---
|
||||
|
||||
# Feishu Doc
|
||||
|
||||
Use the `feishu_doc` tool for Feishu Docx operations.
|
||||
|
||||
If the current environment does not expose `feishu_doc`, tell the user that the required tool is unavailable and stop instead of inventing a workaround.
|
||||
|
||||
## Extract the doc token
|
||||
|
||||
Extract `doc_token` from the document URL.
|
||||
|
||||
Example:
|
||||
|
||||
- `https://xxx.feishu.cn/docx/ABC123def` -> `ABC123def`
|
||||
|
||||
## Start with the right workflow
|
||||
|
||||
Choose the lightest workflow that fits the request.
|
||||
|
||||
### Read a document
|
||||
|
||||
1. Call `{"action":"read","doc_token":"..."}` first.
|
||||
2. Inspect `hint`, `block_types`, and summary fields in the response.
|
||||
3. If the document contains structured content such as tables or images, call `{"action":"list_blocks","doc_token":"..."}`.
|
||||
4. Use `get_block` only when a single block needs closer inspection.
|
||||
|
||||
### Replace or append text content
|
||||
|
||||
Use markdown-oriented actions for plain document content:
|
||||
|
||||
- `write`: replace the entire document
|
||||
- `append`: append content to the end
|
||||
|
||||
Use markdown for headings, lists, code blocks, quotes, links, and images.
|
||||
|
||||
Do not rely on markdown tables. Use table actions instead.
|
||||
|
||||
### Perform block-level edits
|
||||
|
||||
Use these actions when the user wants targeted updates instead of full-document replacement:
|
||||
|
||||
- `list_blocks`
|
||||
- `get_block`
|
||||
- `update_block`
|
||||
- `delete_block`
|
||||
|
||||
Prefer block-level edits when preserving surrounding content matters.
|
||||
|
||||
## Core actions
|
||||
|
||||
### Read
|
||||
|
||||
```json
|
||||
{ "action": "read", "doc_token": "ABC123def" }
|
||||
```
|
||||
|
||||
### Write the whole document
|
||||
|
||||
```json
|
||||
{ "action": "write", "doc_token": "ABC123def", "content": "# Title\n\nMarkdown content..." }
|
||||
```
|
||||
|
||||
### Append content
|
||||
|
||||
```json
|
||||
{ "action": "append", "doc_token": "ABC123def", "content": "Additional content" }
|
||||
```
|
||||
|
||||
### Create a document
|
||||
|
||||
Always pass the requesting user's `open_id` as `owner_open_id` when available so the user automatically gets access to the new document.
|
||||
|
||||
```json
|
||||
{ "action": "create", "title": "New Document", "owner_open_id": "ou_xxx" }
|
||||
```
|
||||
|
||||
With a folder:
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "create",
|
||||
"title": "New Document",
|
||||
"folder_token": "fldcnXXX",
|
||||
"owner_open_id": "ou_xxx"
|
||||
}
|
||||
```
|
||||
|
||||
### Inspect or edit blocks
|
||||
|
||||
```json
|
||||
{ "action": "list_blocks", "doc_token": "ABC123def" }
|
||||
```
|
||||
|
||||
```json
|
||||
{ "action": "get_block", "doc_token": "ABC123def", "block_id": "doxcnXXX" }
|
||||
```
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "update_block",
|
||||
"doc_token": "ABC123def",
|
||||
"block_id": "doxcnXXX",
|
||||
"content": "New text"
|
||||
}
|
||||
```
|
||||
|
||||
```json
|
||||
{ "action": "delete_block", "doc_token": "ABC123def", "block_id": "doxcnXXX" }
|
||||
```
|
||||
|
||||
## Tables
|
||||
|
||||
Use table actions for real Docx tables.
|
||||
|
||||
### Create an empty table
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "create_table",
|
||||
"doc_token": "ABC123def",
|
||||
"row_size": 2,
|
||||
"column_size": 2,
|
||||
"column_width": [200, 200]
|
||||
}
|
||||
```
|
||||
|
||||
Optional: include `parent_block_id` to insert under a specific block.
|
||||
|
||||
### Fill table cells
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "write_table_cells",
|
||||
"doc_token": "ABC123def",
|
||||
"table_block_id": "doxcnTABLE",
|
||||
"values": [
|
||||
["A1", "B1"],
|
||||
["A2", "B2"]
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Create and fill a table in one step
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "create_table_with_values",
|
||||
"doc_token": "ABC123def",
|
||||
"row_size": 2,
|
||||
"column_size": 2,
|
||||
"column_width": [200, 200],
|
||||
"values": [
|
||||
["A1", "B1"],
|
||||
["A2", "B2"]
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Images and file attachments
|
||||
|
||||
### Upload an image
|
||||
|
||||
Use exactly one of `url` or `file_path`.
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "upload_image",
|
||||
"doc_token": "ABC123def",
|
||||
"url": "https://example.com/image.png"
|
||||
}
|
||||
```
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "upload_image",
|
||||
"doc_token": "ABC123def",
|
||||
"file_path": "/tmp/image.png",
|
||||
"parent_block_id": "doxcnParent",
|
||||
"index": 5
|
||||
}
|
||||
```
|
||||
|
||||
For small images, scale the asset before uploading so it displays at a useful size in the document.
|
||||
|
||||
### Upload a file attachment
|
||||
|
||||
Use exactly one of `url` or `file_path`.
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "upload_file",
|
||||
"doc_token": "ABC123def",
|
||||
"url": "https://example.com/report.pdf"
|
||||
}
|
||||
```
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "upload_file",
|
||||
"doc_token": "ABC123def",
|
||||
"file_path": "/tmp/report.pdf",
|
||||
"filename": "Q1-report.pdf"
|
||||
}
|
||||
```
|
||||
|
||||
Optional: include `parent_block_id`.
|
||||
|
||||
## Operating rules
|
||||
|
||||
- Start with `read` for unfamiliar documents.
|
||||
- Escalate to `list_blocks` when `read` indicates structured content.
|
||||
- Use `write` only when replacing the full document is intended.
|
||||
- Prefer block-level edits when the user asks for localized changes.
|
||||
- Use table actions instead of markdown tables.
|
||||
- Preserve the user's access by passing `owner_open_id` on document creation when that identity is available.
|
||||
4
tools/feishu_project/skill/feishu-doc/agents/openai.yaml
Executable file
4
tools/feishu_project/skill/feishu-doc/agents/openai.yaml
Executable file
@@ -0,0 +1,4 @@
|
||||
interface:
|
||||
display_name: "Feishu Doc"
|
||||
short_description: "Read and edit Feishu Docx documents with the feishu_doc tool."
|
||||
default_prompt: "Use the feishu_doc tool to read, create, append, and structurally edit Feishu Docx documents when the user asks about Feishu docs, cloud docs, or docx links."
|
||||
214
tools/feishu_project/skill/feishu-project-issue-data/SKILL.md
Executable file
214
tools/feishu_project/skill/feishu-project-issue-data/SKILL.md
Executable file
@@ -0,0 +1,214 @@
|
||||
---
|
||||
name: feishu-project-issue-data
|
||||
description: Use when working in the yolo26-3d repository and the user wants to read Feishu Project issue views, export a view to structured JSON, classify issue data addresses, download issue data, repair affected standard-path downloads, validate case completeness, or run batch inference on downloaded issue data. Covers fp CLI plus tools/feishu_project/export_feishu_view_issues.py, download_issue_data.py/sh, and run_issue_data_inference.py/sh, including the 董颖-G1Q3 workflow.
|
||||
---
|
||||
|
||||
# Feishu Project Issue Data
|
||||
|
||||
Use the repo dev env for Python commands:
|
||||
|
||||
```bash
|
||||
/root/.codex/skills/use-dongying-dev-env/scripts/with-dev-env.sh python ...
|
||||
```
|
||||
|
||||
or:
|
||||
|
||||
```bash
|
||||
/deeplearning_team/ydong/dongying/miniconda/envs/dev/bin/python ...
|
||||
```
|
||||
|
||||
Current repo defaults are documented in [../../feishu_project.md](../../feishu_project.md).
|
||||
|
||||
## When to use
|
||||
|
||||
- Read or verify a Feishu Project issue view with `fp`
|
||||
- Export a view such as `董颖-G1Q3` to structured JSON
|
||||
- Work with `问题数据地址` / `问题数据地址_PDCL`
|
||||
- Analyze issue tag profiles across `目标` / `问题`
|
||||
- Download issue data into the local workspace
|
||||
- Repair historical bad copies that only copied `sigmastar.1`
|
||||
- Check whether downloaded cases are inference-ready
|
||||
- Batch-run exported-model inference on downloaded issue data
|
||||
|
||||
## Core workflow
|
||||
|
||||
### 1. Verify Feishu access and view contents
|
||||
|
||||
Use `fp` directly when the user wants current data.
|
||||
|
||||
```bash
|
||||
fp view list -p <project_key> -u <user_key> -t issue --name "<view_name>"
|
||||
fp workitem list -o json -p <project_key> -u <user_key> --view "<view_name>" --all
|
||||
fp workitem get <issue_id> -o json -p <project_key> -u <user_key> -t issue
|
||||
```
|
||||
|
||||
### 2. Export structured issue JSON
|
||||
|
||||
Use [../../export_feishu_view_issues.py](../../export_feishu_view_issues.py).
|
||||
|
||||
```bash
|
||||
python ../../export_feishu_view_issues.py \
|
||||
--project-key <project_key> \
|
||||
--user-key <user_key> \
|
||||
--view-name "<view_name>" \
|
||||
--output ../../dongying_g1q3_issue_list.json
|
||||
```
|
||||
|
||||
The export should include:
|
||||
|
||||
- `缺陷标签池`
|
||||
- `问题数据地址`
|
||||
- `问题数据地址_PDCL`
|
||||
- `问题发生frameid`
|
||||
|
||||
### 3. Interpret data-address fields
|
||||
|
||||
Treat these as the same download class:
|
||||
|
||||
- pure `ADAS_...::...` clip references
|
||||
- `mdi raw -r ...` commands
|
||||
|
||||
Use this rule:
|
||||
|
||||
- `ADAS_xxx::yyy` is equivalent to `mdi raw -r ADAS_xxx::yyy -s .`
|
||||
|
||||
Standard paths use these normalization rules:
|
||||
|
||||
- rewrite `hfs/project-G1M3` or `project-G1M3` to `G1M3` when needed
|
||||
- if the path ends with `sigmastar.1`, copy the parent case dir
|
||||
- if the path ends with `sigmastar.1/camera4.bin`, copy the case dir above it
|
||||
- if the copied case has no local `test_data/calibs/camera4.json`, sync a shared parent `test_data` directory when present
|
||||
|
||||
### 4. Analyze issue tag profiles
|
||||
|
||||
Use [../../analyze_issue_tag_profile.py](../../analyze_issue_tag_profile.py) directly, or the G1Q3 wrapper
|
||||
[../../run_issue_tag_profile.sh](../../run_issue_tag_profile.sh).
|
||||
|
||||
```bash
|
||||
bash ../../run_issue_tag_profile.sh
|
||||
```
|
||||
|
||||
Defaults:
|
||||
|
||||
- input JSON: `/data1/dongying/Mono3d/G1Q3/feishu_project/exports/dongying_g1q3_issue_list.json`
|
||||
- report root: `/data1/dongying/Mono3d/G1Q3/feishu_project/reports/issue_tag_profile`
|
||||
|
||||
Output is a single HTML profile report with inline SVG donut charts and `场地问题` / `路测问题` toggle views covering:
|
||||
|
||||
- label completeness and multi-label quality
|
||||
- `目标` / `问题` distributions
|
||||
- `目标 x 问题` matrix
|
||||
- function-domain x problem matrix
|
||||
|
||||
Publish the latest HTML to the static intranet site directory with
|
||||
[../../publish_issue_tag_profile_site.sh](../../publish_issue_tag_profile_site.sh).
|
||||
|
||||
```bash
|
||||
bash ../../publish_issue_tag_profile_site.sh
|
||||
```
|
||||
|
||||
To show or serve through a fixed intranet address, pass `INTRANET_HOST` or
|
||||
`PUBLIC_HOST`:
|
||||
|
||||
```bash
|
||||
INTRANET_HOST=192.168.2.169 bash ../../publish_issue_tag_profile_site.sh
|
||||
INTRANET_HOST=192.168.2.169 bash ../../serve_issue_tag_profile_site.sh restart
|
||||
```
|
||||
|
||||
The address must either belong to the current host or be handled by a reverse
|
||||
proxy/static server at that intranet machine.
|
||||
|
||||
Defaults:
|
||||
|
||||
- site root: `/data1/dongying/Mono3d/G1Q3/feishu_project/site`
|
||||
- site index: `/data1/dongying/Mono3d/G1Q3/feishu_project/site/issue_tag_profile/index.html`
|
||||
- example URL: `http://<intranet-host>:8088/issue_tag_profile/`
|
||||
- Nginx example: [../../nginx_issue_tag_profile.conf.example](../../nginx_issue_tag_profile.conf.example)
|
||||
|
||||
If Nginx is not available yet, start a lightweight static server with
|
||||
[../../serve_issue_tag_profile_site.sh](../../serve_issue_tag_profile_site.sh):
|
||||
|
||||
```bash
|
||||
bash ../../serve_issue_tag_profile_site.sh start
|
||||
```
|
||||
|
||||
### 5. Download or repair issue data
|
||||
|
||||
Use [../../download_issue_data.sh](../../download_issue_data.sh) or [../../download_issue_data.py](../../download_issue_data.py).
|
||||
|
||||
Common modes:
|
||||
|
||||
```bash
|
||||
DRY_RUN=1 bash ../../download_issue_data.sh
|
||||
```
|
||||
|
||||
```bash
|
||||
ONLY_REDOWNLOAD_AFFECTED_CASES=1 bash ../../download_issue_data.sh
|
||||
```
|
||||
|
||||
```bash
|
||||
SKIP_MDI=1 bash ../../download_issue_data.sh --issue-id <id>
|
||||
```
|
||||
|
||||
Defaults:
|
||||
|
||||
- download root: `/data1/dongying/Mono3d/G1Q3/feishu_project/downloaded_issue_data`
|
||||
- manifest: `<download_root>/download_manifest.json`
|
||||
|
||||
Use `ONLY_REDOWNLOAD_AFFECTED_CASES=1` to repair old standard-path copies that previously kept only `sigmastar.1`.
|
||||
|
||||
### 6. Validate inference readiness
|
||||
|
||||
Use the same case-resolution rules as [../../../model_inference/adapters/video_dir_inference_utils.py](../../../model_inference/adapters/video_dir_inference_utils.py).
|
||||
|
||||
A valid case must resolve:
|
||||
|
||||
- `*/sigmastar.1/camera4.bin`
|
||||
- a reachable `camera4.json` from one of:
|
||||
- `case_dir/test_data/calibs/camera4.json`
|
||||
- `case_dir.parent/test_data/calibs/camera4.json`
|
||||
- `case_dir/sigmastar.1/calibs/camera4.json`
|
||||
- `case_dir/calibs/camera4.json`
|
||||
|
||||
Prefer validating with the actual inference path-resolution logic instead of ad hoc file checks.
|
||||
|
||||
### 7. Run batch inference on downloaded issue data
|
||||
|
||||
Use [../../run_issue_data_inference.sh](../../run_issue_data_inference.sh) or [../../run_issue_data_inference.py](../../run_issue_data_inference.py).
|
||||
|
||||
```bash
|
||||
DRY_RUN=1 bash ../../run_issue_data_inference.sh
|
||||
```
|
||||
|
||||
```bash
|
||||
bash ../../run_issue_data_inference.sh
|
||||
```
|
||||
|
||||
Behavior:
|
||||
|
||||
- recursively scans the download root for `*/sigmastar.1/camera4.bin`
|
||||
- calls [../../../model_inference/core/run_two_roi_exported_onnx_infer.py](../../../model_inference/core/run_two_roi_exported_onnx_infer.py) with `--video-case-dir`
|
||||
- mirrors the download-tree relative layout into the inference output root
|
||||
|
||||
Defaults:
|
||||
|
||||
- inference root: `/data1/dongying/Mono3d/G1Q3/feishu_project/inference_issue_data`
|
||||
- manifest: `<inference_root>/inference_manifest.json`
|
||||
|
||||
Useful flags:
|
||||
|
||||
- `SKIP_EXISTING=1`
|
||||
- `ENABLE_ATTR=1`
|
||||
- `SAVE_AGGREGATE_PREDICTIONS=1`
|
||||
- `VIDEO_STRIDE=<n>`
|
||||
- `MAX_IMAGES=<n>`
|
||||
|
||||
## Current repo artifacts
|
||||
|
||||
These files are useful outputs, but they are not the source of truth for latest Feishu data:
|
||||
|
||||
- [../../dongying_g1q3_issue_list.json](../../dongying_g1q3_issue_list.json)
|
||||
- [../../dongying_g1q3_data_address_summary.md](../../dongying_g1q3_data_address_summary.md)
|
||||
- [../../dongying_g1q3_data_address_catalog.md](../../dongying_g1q3_data_address_catalog.md)
|
||||
|
||||
If the user asks for latest status, re-query Feishu with `fp` and regenerate outputs instead of trusting stale local exports.
|
||||
3
tools/feishu_project/start_profile.sh
Executable file
3
tools/feishu_project/start_profile.sh
Executable file
@@ -0,0 +1,3 @@
|
||||
python3 -m http.server 8088 \
|
||||
--bind 0.0.0.0 \
|
||||
--directory /data1/dongying/Mono3d/G1Q3/feishu_project/site
|
||||
164
tools/feishu_project/sync_g1q3_issue_data.sh
Executable file
164
tools/feishu_project/sync_g1q3_issue_data.sh
Executable file
@@ -0,0 +1,164 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
|
||||
PROJECT_ROOT=$(cd "${SCRIPT_DIR}/../.." && pwd)
|
||||
|
||||
PROJECT_KEY=${PROJECT_KEY:-68ef617fb371dc80a10641f7}
|
||||
USER_KEY=${USER_KEY:-7550145433285312514}
|
||||
VIEW_NAME=${VIEW_NAME:-董颖-G1Q3}
|
||||
WORK_ITEM_TYPE=${WORK_ITEM_TYPE:-issue}
|
||||
|
||||
SYNC_ROOT=${SYNC_ROOT:-/data1/dongying/Mono3d/G1Q3/feishu_project}
|
||||
SYNC_MANIFEST_PATH_WAS_SET=${SYNC_MANIFEST_PATH+x}
|
||||
INFERENCE_ROOT_WAS_SET=${INFERENCE_ROOT+x}
|
||||
INFERENCE_MANIFEST_PATH_WAS_SET=${INFERENCE_MANIFEST_PATH+x}
|
||||
EXPORT_JSON=${EXPORT_JSON:-"${SYNC_ROOT}/exports/dongying_g1q3_issue_list.json"}
|
||||
SYNC_MANIFEST_PATH=${SYNC_MANIFEST_PATH:-"${SYNC_ROOT}/exports/dongying_g1q3_issue_list.sync_manifest.json"}
|
||||
SNAPSHOT_DIR=${SNAPSHOT_DIR:-"${SYNC_ROOT}/exports/history"}
|
||||
DOWNLOAD_ROOT=${DOWNLOAD_ROOT:-"${SYNC_ROOT}/downloaded_issue_data"}
|
||||
DOWNLOAD_MANIFEST_PATH=${DOWNLOAD_MANIFEST_PATH:-"${DOWNLOAD_ROOT}/download_manifest.json"}
|
||||
INFERENCE_ROOT=${INFERENCE_ROOT:-"${SYNC_ROOT}/inference_issue_data"}
|
||||
INFERENCE_MANIFEST_PATH=${INFERENCE_MANIFEST_PATH:-"${INFERENCE_ROOT}/inference_manifest.json"}
|
||||
|
||||
PYTHON_BIN=${PYTHON_BIN:-/deeplearning_team/ydong/dongying/miniconda/envs/dev/bin/python}
|
||||
RUN_DOWNLOAD=${RUN_DOWNLOAD:-1}
|
||||
RUN_INFERENCE=${RUN_INFERENCE:-1}
|
||||
ENABLE_TRACKING=${ENABLE_TRACKING:-1}
|
||||
ENABLE_CONVERT=${ENABLE_CONVERT:-1}
|
||||
CONVERT_OUTPUT_DIR_NAME=${CONVERT_OUTPUT_DIR_NAME:-objectlist}
|
||||
CONVERT_TRACKING_JSON_NAME=${CONVERT_TRACKING_JSON_NAME:-merge.json}
|
||||
CONVERT_CAM_ID=${CONVERT_CAM_ID:-}
|
||||
SAVE_SNAPSHOT=${SAVE_SNAPSHOT:-1}
|
||||
REFRESH_CHANGED_ISSUES=${REFRESH_CHANGED_ISSUES:-0}
|
||||
SKIP_EXISTING_INFERENCE=${SKIP_EXISTING_INFERENCE:-1}
|
||||
DRY_RUN=${DRY_RUN:-0}
|
||||
USE_ISSUE_FRAME_WINDOW=${USE_ISSUE_FRAME_WINDOW:-1}
|
||||
FRAME_BEFORE=${FRAME_BEFORE:-200}
|
||||
FRAME_AFTER=${FRAME_AFTER:-200}
|
||||
MISSING_ISSUE_FRAME_POLICY=${MISSING_ISSUE_FRAME_POLICY:-full}
|
||||
ISSUE_ID_MIN=${ISSUE_ID_MIN:-}
|
||||
ISSUE_ID_MAX=${ISSUE_ID_MAX:-6979030662}
|
||||
|
||||
EXPORTED_MODEL=${EXPORTED_MODEL:-${PROJECT_ROOT}/runs/export/train_mono3d_two_roi_20260427-raw_no_edge/merged_model.torchscript}
|
||||
# EXPORTED_MODEL=${EXPORTED_MODEL:-${PROJECT_ROOT}/runs/export/train_mono3d_two_roi_20260506-keep_fake_3d_branch/merged_model.torchscript}
|
||||
|
||||
ISSUE_TRACKING_SCRIPT=${ISSUE_TRACKING_SCRIPT:-"${PROJECT_ROOT}/tools/feishu_project/run_issue_data_tracking.sh"}
|
||||
TRACKING_MODEL_VERSION=${TRACKING_MODEL_VERSION:-20260427}
|
||||
ENABLE_RETEST=${ENABLE_RETEST:-1}
|
||||
RETEST_NAME_KEYWORDS=${RETEST_NAME_KEYWORDS:-AEB,FCW,场地} # AEB,FCW,场地 ACC,HMI,LCC,TSR
|
||||
RETEST_RUN_NAME=${RETEST_RUN_NAME:-$(date +%Y%m%d_%H%M%S)}
|
||||
RETEST_ROOT=${RETEST_ROOT:-"${SYNC_ROOT}/retest_runs/${TRACKING_MODEL_VERSION}_${RETEST_RUN_NAME}"}
|
||||
SHOW_DISTANCE_LABEL=${SHOW_DISTANCE_LABEL:-1}
|
||||
DISTANCE_LABEL_MODE=${DISTANCE_LABEL_MODE:-depth}
|
||||
DISTANCE_LABEL_PANELS=${DISTANCE_LABEL_PANELS:-3d}
|
||||
|
||||
export ENABLE_CONVERT
|
||||
export CONVERT_OUTPUT_DIR_NAME
|
||||
export CONVERT_TRACKING_JSON_NAME
|
||||
export CONVERT_CAM_ID
|
||||
|
||||
if [[ "${ENABLE_RETEST}" == "1" ]]; then
|
||||
if [[ -z "${INFERENCE_ROOT_WAS_SET}" ]]; then
|
||||
INFERENCE_ROOT="${RETEST_ROOT}/inference_issue_data"
|
||||
fi
|
||||
if [[ -z "${INFERENCE_MANIFEST_PATH_WAS_SET}" ]]; then
|
||||
INFERENCE_MANIFEST_PATH="${INFERENCE_ROOT}/inference_manifest.json"
|
||||
fi
|
||||
if [[ -z "${SYNC_MANIFEST_PATH_WAS_SET}" ]]; then
|
||||
SYNC_MANIFEST_PATH="${RETEST_ROOT}/dongying_g1q3_issue_list.sync_manifest.json"
|
||||
fi
|
||||
fi
|
||||
|
||||
CMD=(
|
||||
"${PYTHON_BIN}" "${PROJECT_ROOT}/tools/feishu_project/sync_issue_data.py"
|
||||
--project-key "${PROJECT_KEY}"
|
||||
--user-key "${USER_KEY}"
|
||||
--view-name "${VIEW_NAME}"
|
||||
--work-item-type "${WORK_ITEM_TYPE}"
|
||||
--output-json "${EXPORT_JSON}"
|
||||
--sync-manifest-path "${SYNC_MANIFEST_PATH}"
|
||||
--snapshot-dir "${SNAPSHOT_DIR}"
|
||||
--python-bin "${PYTHON_BIN}"
|
||||
--download-root "${DOWNLOAD_ROOT}"
|
||||
--download-manifest-path "${DOWNLOAD_MANIFEST_PATH}"
|
||||
--inference-root "${INFERENCE_ROOT}"
|
||||
--inference-manifest-path "${INFERENCE_MANIFEST_PATH}"
|
||||
)
|
||||
|
||||
if [[ "${RUN_DOWNLOAD}" == "1" ]]; then
|
||||
CMD+=(--run-download)
|
||||
fi
|
||||
|
||||
if [[ "${RUN_INFERENCE}" == "1" ]]; then
|
||||
CMD+=(--run-inference)
|
||||
fi
|
||||
|
||||
if [[ "${USE_ISSUE_FRAME_WINDOW}" == "1" ]]; then
|
||||
CMD+=(--use-issue-frame-window --frame-before "${FRAME_BEFORE}" --frame-after "${FRAME_AFTER}" --missing-issue-frame-policy "${MISSING_ISSUE_FRAME_POLICY}")
|
||||
fi
|
||||
|
||||
if [[ "${ENABLE_TRACKING}" == "1" ]]; then
|
||||
CMD+=(--run-tracking --issue-tracking-script "${ISSUE_TRACKING_SCRIPT}")
|
||||
fi
|
||||
|
||||
if [[ "${SAVE_SNAPSHOT}" == "1" ]]; then
|
||||
CMD+=(--save-snapshot)
|
||||
fi
|
||||
|
||||
if [[ "${REFRESH_CHANGED_ISSUES}" == "1" ]]; then
|
||||
CMD+=(--refresh-changed-issues)
|
||||
fi
|
||||
|
||||
if [[ "${SKIP_EXISTING_INFERENCE}" == "1" ]]; then
|
||||
CMD+=(--skip-existing-inference)
|
||||
fi
|
||||
|
||||
if [[ "${DRY_RUN}" == "1" ]]; then
|
||||
CMD+=(--dry-run)
|
||||
fi
|
||||
|
||||
if [[ -n "${ISSUE_ID_MIN}" ]]; then
|
||||
CMD+=(--issue-id-min "${ISSUE_ID_MIN}")
|
||||
fi
|
||||
|
||||
if [[ -n "${ISSUE_ID_MAX}" ]]; then
|
||||
CMD+=(--issue-id-max "${ISSUE_ID_MAX}")
|
||||
fi
|
||||
|
||||
if [[ -n "${EXPORTED_MODEL}" ]]; then
|
||||
CMD+=(--exported-model "${EXPORTED_MODEL}")
|
||||
fi
|
||||
|
||||
if [[ -n "${TRACKING_MODEL_VERSION}" ]]; then
|
||||
CMD+=(--tracking-model-version "${TRACKING_MODEL_VERSION}")
|
||||
fi
|
||||
|
||||
if [[ "${SHOW_DISTANCE_LABEL}" == "1" ]]; then
|
||||
CMD+=("--inference-arg=--show-distance-label")
|
||||
if [[ -n "${DISTANCE_LABEL_MODE}" ]]; then
|
||||
CMD+=("--inference-arg=--distance-label-mode" "--inference-arg=${DISTANCE_LABEL_MODE}")
|
||||
fi
|
||||
if [[ -n "${DISTANCE_LABEL_PANELS}" ]]; then
|
||||
# shellcheck disable=SC2206
|
||||
DISTANCE_LABEL_PANELS_ARR=(${DISTANCE_LABEL_PANELS})
|
||||
CMD+=("--inference-arg=--distance-label-panels")
|
||||
for panel in "${DISTANCE_LABEL_PANELS_ARR[@]}"; do
|
||||
CMD+=("--inference-arg=${panel}")
|
||||
done
|
||||
fi
|
||||
fi
|
||||
|
||||
if [[ "${ENABLE_RETEST}" == "1" ]] && [[ -n "${RETEST_NAME_KEYWORDS}" ]]; then
|
||||
IFS=',' read -r -a RETEST_KEYWORD_ARRAY <<< "${RETEST_NAME_KEYWORDS}"
|
||||
for keyword in "${RETEST_KEYWORD_ARRAY[@]}"; do
|
||||
keyword="${keyword#"${keyword%%[![:space:]]*}"}"
|
||||
keyword="${keyword%"${keyword##*[![:space:]]}"}"
|
||||
if [[ -n "${keyword}" ]]; then
|
||||
CMD+=(--issue-name-keyword "${keyword}")
|
||||
fi
|
||||
done
|
||||
fi
|
||||
|
||||
CMD+=("$@")
|
||||
"${CMD[@]}"
|
||||
1582
tools/feishu_project/sync_issue_data.py
Executable file
1582
tools/feishu_project/sync_issue_data.py
Executable file
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user