Repository navigation
feat: separate image loader - #29712
fanshuaishuai1 wants to merge 27 commits into
Conversation
Port the multiprocess overlapped image-loading pipeline onto the current EncoderScheduler/MMEncoder structure: a pool of loader processes decode each collected IMAGE batch off the GPU path while a separate event-loop thread runs the GPU encode, so the next batch loads while the current one encodes. - Add image_loader.py (torch.multiprocessing worker pool) - EncoderScheduler spawns the loader pool + preprocess thread; IMAGE groups are enqueued for out-of-process decode, AUDIO/VIDEO stay on the inline path - batch_encode accepts preloaded images (rank-0 only); TP workers reload inline so collective broadcasts stay aligned - Register SGLANG_ENCODER_IMAGE_LOADER_WORKERS in environ.py Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
|
@fanshuaishuai1 Please resolve the conflicts. |
|
@fanshuaishuai1 Please fix the lint |
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
LGTM, let's see the CI status. |
|
It seems that only TP=1 is currently supported. Should we support TP > 1 in this PR, or leave it for a follow-up? |
leave it for a follow-up |
|
@fanshuaishuai1 In that case, we should raise an error when TP > 1. |
Images are preloaded on rank 0 only while TP workers load inline, and the loader-consumer thread dispatches on a second event loop, so the ZMQ send order can diverge from rank-0's collective forward order. Fail fast at startup instead of risking a collective mismatch. The DP worker path is unaffected since it forces tp_size=1. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Done |
|
We need an option to turn this feature on. |
Add SGLANG_ENCODER_ENABLE_IMAGE_LOADER (EnvBool, default on) to gate the multiprocess overlapped image-loading pipeline. When disabled, the loader process pool and preprocess/encode thread are not spawned and IMAGE batches decode inline on the dispatch path. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
@fanshuaishuai1 Please stop merge main, this will trigger the CI limits |
| coalesce_same_turn: bool = False, | ||
| request_timeout: float = ENCODER_REQ_TIMEOUT, | ||
| ): | ||
| if encoder.server_args.tp_size > 1: |
There was a problem hiding this comment.
I'm not sure if this will break the current logic (e.g., cases where the encoder is intended to run with TP > 1). I'd like to make this feature configurable via an environment variable.
get |
|
Thanks @fanshuaishuai1. Closing this because it has had no updates in 91 days. Reopen it if the work is still relevant. Some directories moved recently, so an older branch may need retargeting: |
Port the multiprocess overlapped image-loading pipeline onto the current EncoderScheduler/MMEncoder structure: a pool of loader processes decode each collected IMAGE batch off the GPU path while a separate event-loop thread runs the GPU encode, so the next batch loads while the current one encodes.
Motivation
Modifications
Accuracy Tests
Speed Tests and Profiling
Checklist
Review and Merge Process
/tag-and-rerun-ci,/tag-run-ci-label,/rerun-failed-ciCI States
Latest PR Test (Base): ❌ Run #30974500404
Latest PR Test (Extra): ❌ Run #30974500205