STORRE
STORRE: Stirling Online Research Repository
http://hdl.handle.net/1893/35519| Appears in Collections: | Computing Science and Mathematics Conference Papers and Proceedings |
| Author(s): | Sobania, Dominik Geiger, Alina Callan, James Brownlee, Alexander Hanna, Carol Moussa, Rebecca Zamorano López, Mar Petke, Justyna Sarro, Federica |
| Contact Email: | alexander.brownlee@stir.ac.uk |
| Title: | Evaluating Explanations for Software Patches Generated by Large Language Models |
| Citation: | Sobania D, Geiger A, Callan J, Brownlee A, Hanna C, Moussa R, Zamorano López M, Petke J & Sarro F (2023) Evaluating Explanations for Software Patches Generated by Large Language Models. In: Symposium on Search-Based Software Engineering- Challenge Track, San Francisco, CA, USA, 08.12.2023-08.12.2023. |
| Date Deposited: | 17-Oct-2023 |
| Conference Name: | Symposium on Search-Based Software Engineering- Challenge Track |
| Conference Dates: | 2023-12-08 - 2023-12-08 |
| Conference Location: | San Francisco, CA, USA |
| Abstract: | Large language models (LLMs) have recently been integrated in a variety of applications including software engineering tasks. In this work, we study the use of LLMs to enhance the explainability of software patches. In particular, we evaluate the performance of GPT 3.5 in explaining patches generated by the search-based automated program repair system ARJA-e for 30 bugs from the popular Defects4J benchmark. We also investigate the performance achieved when explaining the corresponding patches written by software developers. We find that on average 84% of the LLM explanations for machine-generated patches were correct and 54% were complete for the studied categories in at least 1 out of 3 runs. Furthermore, we find that the LLM generates more accurate explanations for machine-generated patches than for human-written ones. |
| Status: | AM - Accepted Manuscript |
| Rights: | This item has been embargoed for a period. During the embargo please use the Request a Copy feature at the foot of the Repository record to request a copy directly from the author. You can only request a copy if you wish to use this work for your own research or private study. |
| Licence URL(s): | http://creativecommons.org/licenses/by/4.0/ |
| File | Description | Size | Format | |
|---|---|---|---|---|
| ssbse23challenge-final19.pdf | Fulltext - Accepted Version | 295.27 kB | Adobe PDF | Under Embargo until 2025-09-30 Request a copy |
Note: If any of the files in this item are currently embargoed, you can request a copy directly from the author by clicking the padlock icon above. However, this facility is dependent on the depositor still being contactable at their original email address.
This item is protected by original copyright |
A file in this item is licensed under a Creative Commons License
Items in the Repository are protected by copyright, with all rights reserved, unless otherwise indicated.
The metadata of the records in the Repository are available under the CC0 public domain dedication: No Rights Reserved https://creativecommons.org/publicdomain/zero/1.0/
If you believe that any material held in STORRE infringes copyright, please contact library@stir.ac.uk providing details and we will remove the Work from public display in STORRE and investigate your claim.