Image-splitting: Enabling higher resolutions through image-splitting

[AdonaiVera]

I'm planning to work on an image-splitting feature to better support high-resolution inputs. Before diving in, I wanted to share my approach and get your thoughts to make sure it aligns with the repo's design.

My current plan:

• Move the image-splitting logic to the collator, so we can process batches of images (instead of handling them individually in the dataset class).
• Update the image processor to accept a list of images and apply do_image_splitting as a param.
• After resizing images for the vision encoder (e.g., max edge = 224), the processor would:
  • Split the images into patches in batch.
  • Include the original image in the batch for reference.
• In the collator, after patching:
  I checked how smolVLM expand prompts using rows/cols and image tokens and noticed that it includes special tokens like <image>, <fake>, and <global>. I was planning to follow a similar approach by adding these tokens directly in the collator, once the image patches are ready. And ensuring that the dataset-level prompt also includes a placeholder for the image, so the collator knows where to inject the token(s).

A few questions:

1. Does it make sense to include the splitting logic directly in the collator, or would you recommend handling it elsewhere?
2. Is it okay to expand prompts using rows/cols and image tokens after patching (in the collator)?
3. Would it be okay to insert the <image>, <fake>, and <global> tokens directly in the collator after patching (like smolVLM), and just ensure that the dataset prompt includes a placeholder so the collator knows where to inject them?

Happy to adjust the plan based on your feedback! Just wanted to double-check before I start pushing changes.

Thanks in advance 🙏

[lusxvr]

Great plan! This is something we also wanted to tackle next, since one of our goals for nanoVLM is to enable us to research how to improve the image tokenization / handling of high resolution images.

I think before we tackle image splitting, we need the ability to combine text and images more freely. Currently we simply concatenate the image and text embeddings. In my opinion, the first step would be to move from this to the introduction of a <img_start> and <img_end> token or multiple dummy <img> tokens that get replaced with the actual tokens later on.

[lusxvr]

After resizing images for the vision encoder (e.g., max edge = 224), the processor would:

• Split the images into patches in batch.
• Include the original image in the batch for reference.

I feel like it makes more sense to instead of resizing the original image to 224 and the splitting it, to split the high resolution image into multiple patches and scale each patch to 224 along with the original image scaled to 224

[AdonaiVera]

I feel like it makes more sense to instead of resizing the original image to 224 and the splitting it, to split the high resolution image into multiple patches and scale each patch to 224 along with the original image scaled to 224

Yes, you're right. What I meant with the resizing is that I’m trying to replicate how SmolVLM handles it, where images are resized to multiples of the vision_encoder_max_size (224 in our case) while preserving the aspect ratio.

I think before we tackle image splitting, we need the ability to combine text and images more freely. Currently we simply concatenate the image and text embeddings. In my opinion, the first step would be to move from this to the introduction of a <img_start> and <img_end> token or multiple dummy <img> tokens that get replaced with the actual tokens later on.

I can definitely start working on this. I was also thinking about improving how we handle the tokens. Do you think it would make sense to adopt the approach from SmolVLM with <fake_token_around_image>, <global_image_token>, and <image_token>? It might add flexibility and make the integration between text and image more seamless.

[lusxvr]

Yes, I think we should definitely adopt the SmolVLM approach of using additional tokens to specify where the image should go!

[lusxvr]

Started this in #82

[AdonaiVera]

Wow, I was already working on this too! This weekend I’ll have more free time, so I’ll check it and try to add some suggestions or improvements if it’s not merged yet. Thank you @lusxvr! 🙌