Please use this identifier to cite or link to this item:
http://hdl.handle.net/1893/36839
Appears in Collections: | Computing Science and Mathematics Journal Articles |
Peer Review Status: | Refereed |
Title: | Large Language Model Based Mutations in Genetic Improvement |
Author(s): | Brownlee, Alexander E I Callan, James Even-Mendoza, Karine Geiger, Alina Hanna, Carol Petke, Justyna Sarro, Federica Sobania, Dominik |
Contact Email: | alexander.brownlee@stir.ac.uk |
Keywords: | Large language models Genetic imporvement |
Issue Date: | 21-Jan-2025 |
Date Deposited: | 8-Oct-2024 |
Citation: | Brownlee AEI, Callan J, Even-Mendoza K, Geiger A, Hanna C, Petke J, Sarro F & Sobania D (2025) Large Language Model Based Mutations in Genetic Improvement. <i>Automated Software Engineering</i>, 32 (15). https://doi.org/10.1007/s10515-024-00473-6 |
Abstract: | Ever since the first large language models (LLMs) have become available, both academics and practitioners have used them to aid software engineering tasks. However, little research as yet has been done in combining search-based software engineering (SBSE) and LLMs. In this paper, we evaluate the use of LLMs as mutation operators for genetic improvement (GI), an SBSE approach, to improve the GI search process. In a preliminary work, we explored the feasibility of combining the Gin Java GI toolkit with OpenAI LLMs in order to generate an edit for the JCodec tool. Here we extend this investigation involving three LLMs and three types of prompt, and five real-world software projects. We sample the edits at random, as well as using local search. We also conducted a qualitative analysis to understand why LLM-generated code edits break as part of our evaluation. Our results show that, compared with conventional statement GI edits, LLMs produce fewer unique edits, but these compile and pass tests more often, with the OpenAI model finding test-passing edits 77% of the time. The OpenAI and Mistral LLMs are roughly equal in finding the best run-time improvements. Simpler prompts are more successful than those providing more context and examples. The qualitative analysis reveals a wide variety of areas where LLMs typically fail to produce valid edits commonly including inconsistent formatting, generating non-Java syntax, or refusing to provide a solution. |
DOI Link: | 10.1007/s10515-024-00473-6 |
Rights: | This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. |
Licence URL(s): | http://creativecommons.org/licenses/by/4.0/ |
Files in This Item:
File | Description | Size | Format | |
---|---|---|---|---|
s10515-024-00473-6.pdf | Fulltext - Published Version | 2.17 MB | Adobe PDF | View/Open |
This item is protected by original copyright |
A file in this item is licensed under a Creative Commons License
Items in the Repository are protected by copyright, with all rights reserved, unless otherwise indicated.
The metadata of the records in the Repository are available under the CC0 public domain dedication: No Rights Reserved https://creativecommons.org/publicdomain/zero/1.0/
If you believe that any material held in STORRE infringes copyright, please contact library@stir.ac.uk providing details and we will remove the Work from public display in STORRE and investigate your claim.