Skip to content

Conversation

Nabarup1
Copy link

This pull request introduces support for finetuning the T2U (Text-to-Unit) component in SeamlessM4T_v2_large models that utilize the UnitYNART2UModel, which was previously not supported in the official finetuning trainer.py. The UnitYFinetuneWrapper has been extended to detect and handle both UnitYT2UModel (v1) and UnitYNART2UModel (v2), enabling comprehensive T2U finetuning for v2 models. Additionally, the dummy_context has been replaced with nullcontext() to ensure robust context management and prevent attribute errors during training. These changes ensure that users can now reliably finetune the T2U module in SeamlessM4T_v2_large without encountering runtime errors or missing model support.

What has been done:

  • Added explicit support for UnitYNART2UModel in the T2U finetuning workflow.
  • Updated the forward logic to accommodate the specific interface and output shape of UnitYNART2UModel.
  • Replaced the use of dummy_context with nullcontext() throughout the trainer to prevent context-related attribute errors.
  • Improved error messaging for unsupported T2U model types, clarifying which models are supported for finetuning[^1].

…); use nullcontext for dummy_context in trainer.py

This update enables finetuning of the T2U component for SeamlessM4T_v2_large models that use the UnitYNART2UModel, which was previously unsupported in the official trainer. The UnitYFinetuneWrapper now detects and handles UnitYNART2UModel in addition to UnitYT2UModel, allowing T2U finetuning for v2 models. Additionally, dummy_context has been replaced with nullcontext() to prevent attribute errors during context management. These changes make it possible to run T2U finetuning workflows for SeamlessM4T_v2_large without runtime errors or missing model support.
@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jun 30, 2025
@Nabarup1
Copy link
Author

Nabarup1 commented Jul 1, 2025

@samueljamesbell @Celebio @mhlakhani
Kindly requesting your review on this PR, which introduces T2U finetuning support for UnitYNART2UModel (SeamlessM4T_v2_large) in trainer.py and updates context handling for improved reliability. Your insights and feedback would be highly valuable.

@Omaralsaabi
Copy link

Thank you, that saved me a lot of time, but I made a change in line 334 to make it work.
find_unused = self.params.finetune_mode in [FinetuneMode.TEXT_TO_SPEECH, FinetuneMode.SPEECH_TO_SPEECH]

@Nabarup1
Copy link
Author

Nabarup1 commented Sep 7, 2025

Appreciate you testing this out, @Omaralsaabi — thanks for pointing out the find_unused tweak. Makes sense. I can fold that into this PR so others don’t run into the same issue.

@Nabarup1
Copy link
Author

Nabarup1 commented Sep 7, 2025

Hey @samueljamesbell @Celebio @mhlakhani — just circling back on this. Checks are all green, and with @Omaralsaabi's feedback addressed it should be good to go. Would be great to get a review/approval when you get a chance 🙏

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants