start_server.sh clones pyworker into /workspace/vast-pyworker after the
provisioning phase has run, so a provisioning script that wants to ship
a custom benchmark workflow cannot write to misc/benchmark.json — that
path doesn't exist yet at provisioning time, and pre-creating it would
make the subsequent clone fail.
Allow provisioning to drop the workflow anywhere (e.g. /workspace) and
point the worker at it via the BENCHMARK_JSON_PATH env var. The in-tree
file still takes precedence (so forks with a baked-in benchmark keep
working unchanged); the env var is consulted only as a second choice,
and a misconfigured path logs a warning rather than silently degrading
to the SD1.5 fallback.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The "Use PyWorker SDK" rewrite (4380d98) replaced the dynamic
ComfyWorkflowData.for_test() benchmark logic with a hardcoded list of 11
SD1.5 Text2Image payloads, dropped misc/benchmark.json.example and
misc/test_prompts.txt, and stopped honouring the BENCHMARK_TEST_*
environment variables. The README's documented behaviour (custom
workflow via benchmark.json, env-var-tuned fallback) had no
implementation behind it.
Restore the original two-tier behaviour against the new SDK by passing
BenchmarkConfig(generator=make_benchmark_payload) instead of a static
dataset, splitting the load logic into a custom-workflow path and a
fallback path, and re-shipping the misc/ assets.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Rob Ballantyne