# Train T5 model for commit message generation **URL:** \ **Category:** Flax/JAX Projects\ **Created:** [June 24, 2021, 8:42am UTC](https://discuss.huggingface.co/t/train-t5-model-for-commit-message-generation/7231 "2021-06-24T08:42:26Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![mozharovsky](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mozharovsky/32/1480_2.png) [@mozharovsky](https://discuss.huggingface.co/u/mozharovsky)\ **Post date:** [June 24, 2021, 8:42am UTC](https://discuss.huggingface.co/t/train-t5-model-for-commit-message-generation/7231/1 "2021-06-24T08:42:26Z")
# T5 for commit message generation Commit message generation is known to be a difficult task due to the following reasons: 1. No available clean and large dataset for pre-training and fine-tuning 2. (Almost) No available pre-trained seq2seq models on both programming and natural languages corpora 3. Context limitations (sequence length, external knowledge, etc.) In the recent research [[1](https://arxiv.org/pdf/2104.02443.pdf)] [[2](https://arxiv.org/pdf/2105.08645.pdf)] [[3](https://arxiv.org/pdf/2103.06333.pdf)] it has been shown that pre-trained transformer models (like T5 or BART) can capture code understanding domain and fine-tune for various seq2seq programming downstream tasks (e.g. code summarization, code documentation generation, commit message generation, etc). Nevertheless, each of the listed downstream tasks deserves more thorough research. I’ve chosen commit message generation to be such a task because it has many practical applications, such as auto version control systems. To wrap up, these are the main goals of this research project: 1. Release a publicly available dataset for programming language models pre-training 2. Release a publicly available dataset for commit message generation fine-tuning 3. Release a publicly available (pre)-trained T5 model for commit message generation 4. Release all pre-processing, post-processing, and training scripts for further research ## 2. Language We’ll use English and Python languages for training. This research is limited in time, so it would be great to show a strong baseline for at least one pair of languages. Later we’ll be able to extend the model for more language pairs. ## 3. Model We’ll be using a random T5 model similar to `t5-base` in configuration. ## 4. Datasets Possible links to publicly available datasets include: - [bigquery-public-data:github\_repos](https://console.cloud.google.com/bigquery) - [C4](https://huggingface.co/datasets/c4) Please note that the BigQuery GitHub dump needs to be cleaned up before being used for pre-training/fine-tuning. We should also consider using only those repos whose license allows us to do so. ## 5. Training scripts We can make use of [run\_summarization\_flax.py](https://github.com/huggingface/transformers/blob/master/examples/flax/summarization/run_summarization_flax.py) to train the model. ## 6. (Optional) Challenges The main challenges are to prepare the dataset and set up training tasks in Flax. Besides, further challenges include facing the short context sizes and transferring knowledge across domains. I suppose we could try using sparse attention blocks and techniques like RAG for bringing the domain knowledge. ## 7. (Optional) Desired project outcome The desired project outcome is to achieve the listed goals and have fun! A demo would be a simple application (e.g., VSCode extension) that generates commit messages given the changed python files. ## 8. (Optional) Reads The following links can be useful to better understand the project and what has previously been done in the research community. - [Contextualized Code Representation Learning for Commit Message Generation](https://arxiv.org/pdf/2007.06934.pdf) - [A Transformer-based Approach for Source Code Summarization](https://arxiv.org/pdf/2005.00653.pdf) - [PYMT5: multi-mode translation of natural language and PYTHON code with transformers](https://arxiv.org/pdf/2010.03150.pdf) - [Unsupervised Translation of Programming Languages](https://arxiv.org/pdf/2006.03511.pdf) ---
**Author:** ![bhavnicksm](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bhavnicksm/32/3547_2.png) [@bhavnicksm](https://discuss.huggingface.co/u/bhavnicksm)\ **Post date:** [June 24, 2021, 11:45am UTC](https://discuss.huggingface.co/t/train-t5-model-for-commit-message-generation/7231/2 "2021-06-24T11:45:41Z")
This idea is definitely quite ✨ interesting ✨ I would love to join in and be a part of this 🤗 I have worked on Seq-2-Seq tasks like Machine Translation on low-resource languages, Character Transliteration for code-mixed conversations, Built and Trained Transformer models in both PyTorch as well as TensorFlow before. I have been meaning to find another seq-2-seq task to work on and this is just ✨ ideal ✨. ---
**Author:** ![mozharovsky](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mozharovsky/32/1480_2.png) [@mozharovsky](https://discuss.huggingface.co/u/mozharovsky)\ **Post date:** [June 24, 2021, 2:44pm UTC](https://discuss.huggingface.co/t/train-t5-model-for-commit-message-generation/7231/3 "2021-06-24T14:44:25Z")
Hey, thanks for being interested in the project! I’ll be super happy to see you on board! 🤗 ---
**Author:** ![gagan3012](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/gagan3012/32/1317_2.png) [@gagan3012](https://discuss.huggingface.co/u/gagan3012)\ **Post date:** [June 24, 2021, 4:16pm UTC](https://discuss.huggingface.co/t/train-t5-model-for-commit-message-generation/7231/4 "2021-06-24T16:16:36Z")
This seems very interesting can I please join this team? ---
**Author:** ![mozharovsky](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mozharovsky/32/1480_2.png) [@mozharovsky](https://discuss.huggingface.co/u/mozharovsky)\ **Post date:** [June 24, 2021, 6:43pm UTC](https://discuss.huggingface.co/t/train-t5-model-for-commit-message-generation/7231/5 "2021-06-24T18:43:24Z")
Hey! Sure, welcome on board! 🤗 ---
**Author:** ![mozharovsky](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mozharovsky/32/1480_2.png) [@mozharovsky](https://discuss.huggingface.co/u/mozharovsky)\ **Post date:** [June 24, 2021, 6:46pm UTC](https://discuss.huggingface.co/t/train-t5-model-for-commit-message-generation/7231/6 "2021-06-24T18:46:40Z")
Does anybody know TypeScript btw? 😄 Looking right now at VSCode Extensions API – it’s awesome! I’m sure we’ll be able to integrate the model into the commit message generation pipeline within an extension. ---
**Author:** ![RastSD](https://avatars.discourse-cdn.com/v4/letter/r/9dc877/32.png) [@RastSD](https://discuss.huggingface.co/u/RastSD)\ **Post date:** [June 25, 2021, 8:44am UTC](https://discuss.huggingface.co/t/train-t5-model-for-commit-message-generation/7231/9 "2021-06-25T08:44:42Z")
I’m very interested and would like to join as well 😊 ---
**Author:** ![agemagician](https://avatars.discourse-cdn.com/v4/letter/a/d07c76/32.png) [@agemagician](https://discuss.huggingface.co/u/agemagician)\ **Post date:** [June 25, 2021, 11:41am UTC](https://discuss.huggingface.co/t/train-t5-model-for-commit-message-generation/7231/10 "2021-06-25T11:41:05Z")
There are already T5 models that was trained for this task: > **[Models - Hugging Face](https://huggingface.co/models?search=commit)** > > We’re on a journey to advance and democratize artificial intelligence through open source and open science. This repo should help you to extend the pretrained CodeTrans models for more tasks or languages: > **[GitHub - agemagician/CodeTrans: Pretrained Language Models for Source code](https://github.com/agemagician/CodeTrans)** > > Pretrained Language Models for Source code. Contribute to agemagician/CodeTrans development by creating an account on GitHub. Good luck. ---
**Author:** ![patrickvonplaten](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/patrickvonplaten/32/2171_2.png) [@patrickvonplaten](https://discuss.huggingface.co/u/patrickvonplaten)\ **Post date:** [June 25, 2021, 5:25pm UTC](https://discuss.huggingface.co/t/train-t5-model-for-commit-message-generation/7231/11 "2021-06-25T17:25:36Z")
That’s a great project description - excited about the results! ---
**Author:** ![bhavnicksm](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bhavnicksm/32/3547_2.png) [@bhavnicksm](https://discuss.huggingface.co/u/bhavnicksm)\ **Post date:** [June 25, 2021, 6:07pm UTC](https://discuss.huggingface.co/t/train-t5-model-for-commit-message-generation/7231/12 "2021-06-25T18:07:38Z")
Hey @agemagician! 🤗 Thanks for pointing out great resources for this task. ❤ I noticed that the paper CodeTrans as well as the Git Commit Message Generator model available on HuggingFace Hub is only trained for **Java**. I think since Java and Python have fundamentally different languages with different synatctical structure and semantics, fine-tuning that model may not yeild the best results. Though I would not mind trying at all and comparing the results, if we have time. ---
**Author:** ![agemagician](https://avatars.discourse-cdn.com/v4/letter/a/d07c76/32.png) [@agemagician](https://discuss.huggingface.co/u/agemagician)\ **Post date:** [June 25, 2021, 8:02pm UTC](https://discuss.huggingface.co/t/train-t5-model-for-commit-message-generation/7231/13 "2021-06-25T20:02:27Z")
ProtTrans actually has models which were trained on 9 programming languages + English language using self-supervised learning. You can easily fine-tune them on any task and almost all famous programming languages. For example: > **[SEBIS/code\_trans\_t5\_large\_transfer\_learning\_pretrain · Hugging Face](https://huggingface.co/SEBIS/code_trans_t5_large_transfer_learning_pretrain)** > > We’re on a journey to advance and democratize artificial intelligence through open source and open science. > **[SEBIS/code\_trans\_t5\_small\_transfer\_learning\_pretrain · Hugging Face](https://huggingface.co/SEBIS/code_trans_t5_small_transfer_learning_pretrain)** > > We’re on a journey to advance and democratize artificial intelligence through open source and open science. > **[SEBIS/code\_trans\_t5\_base\_transfer\_learning\_pretrain · Hugging Face](https://huggingface.co/SEBIS/code_trans_t5_base_transfer_learning_pretrain)** > > We’re on a journey to advance and democratize artificial intelligence through open source and open science. ---
**Author:** ![mozharovsky](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mozharovsky/32/1480_2.png) [@mozharovsky](https://discuss.huggingface.co/u/mozharovsky)\ **Post date:** [June 25, 2021, 9:44pm UTC](https://discuss.huggingface.co/t/train-t5-model-for-commit-message-generation/7231/14 "2021-06-25T21:44:51Z")
Thanks, @patrickvonplaten! Shall we consider forming two teams under this project? We have both pre-training and fine-tuning research experiments that can be parallelized across the teams. ---
**Author:** ![mozharovsky](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mozharovsky/32/1480_2.png) [@mozharovsky](https://discuss.huggingface.co/u/mozharovsky)\ **Post date:** [June 25, 2021, 10:19pm UTC](https://discuss.huggingface.co/t/train-t5-model-for-commit-message-generation/7231/15 "2021-06-25T22:19:36Z")
Hey @agemagician, CodeTrans is a great research project! Our research is highly backed by their results. Anyway, we still have space for improving their ideas to make pre-trained models stronger domain learners. Besides, we’re missing publicly available high-quality data and benchmarks for us and other researchers to improve upon existing ideas and bring new ones. We believe that the HuggingFace ecosystem is a perfect place to make research open for all, and hopefully, our research can enforce these values further. 🤗 Regards, Eugene ---
**Author:** ![patrickvonplaten](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/patrickvonplaten/32/2171_2.png) [@patrickvonplaten](https://discuss.huggingface.co/u/patrickvonplaten)\ **Post date:** [June 29, 2021, 2:17pm UTC](https://discuss.huggingface.co/t/train-t5-model-for-commit-message-generation/7231/16 "2021-06-29T14:17:27Z")
Very exciting project - putting it down 🙂 Since one team is good here! Since you are more than 5, we might have 2 TPUs for you! ---
**Author:** ![mozharovsky](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mozharovsky/32/1480_2.png) [@mozharovsky](https://discuss.huggingface.co/u/mozharovsky)\ **Post date:** [June 29, 2021, 4:08pm UTC](https://discuss.huggingface.co/t/train-t5-model-for-commit-message-generation/7231/17 "2021-06-29T16:08:51Z")
Amazing news, thank you! Just a quick question – can we add more participants since we have a few more slots available? ---
**Author:** ![bharat-raghunathan](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bharat-raghunathan/32/4726_2.png) [@bharat-raghunathan](https://discuss.huggingface.co/u/bharat-raghunathan)\ **Post date:** [June 29, 2021, 5:51pm UTC](https://discuss.huggingface.co/t/train-t5-model-for-commit-message-generation/7231/18 "2021-06-29T17:51:52Z")
Hi, I would like to join this team if spots are still available! I have some experience in Software Engineering and GitHub APIs that may help ---
**Author:** ![mozharovsky](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mozharovsky/32/1480_2.png) [@mozharovsky](https://discuss.huggingface.co/u/mozharovsky)\ **Post date:** [June 30, 2021, 8:32am UTC](https://discuss.huggingface.co/t/train-t5-model-for-commit-message-generation/7231/19 "2021-06-30T08:32:25Z")
Hey, join us on our [Discord channel](https://discord.gg/QHKgksmP)! 👋 ---
**Author:** ![patrickvonplaten](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/patrickvonplaten/32/2171_2.png) [@patrickvonplaten](https://discuss.huggingface.co/u/patrickvonplaten)\ **Post date:** [June 30, 2021, 12:35pm UTC](https://discuss.huggingface.co/t/train-t5-model-for-commit-message-generation/7231/20 "2021-06-30T12:35:49Z")
Yes! This is fine 🙂 ---
**Author:** ![patrickvonplaten](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/patrickvonplaten/32/2171_2.png) [@patrickvonplaten](https://discuss.huggingface.co/u/patrickvonplaten)\ **Post date:** [July 1, 2021, 10:01am UTC](https://discuss.huggingface.co/t/train-t5-model-for-commit-message-generation/7231/22 "2021-07-01T10:01:04Z")
added you 🙂 ---
**Author:** ![mozharovsky](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mozharovsky/32/1480_2.png) [@mozharovsky](https://discuss.huggingface.co/u/mozharovsky)\ **Post date:** [July 2, 2021, 11:03am UTC](https://discuss.huggingface.co/t/train-t5-model-for-commit-message-generation/7231/24 "2021-07-02T11:03:47Z")
Hey @jxuhf, welcome on board! Please, join our Discord channel (#t5-commit-message).