Berger, Harel

Dakhama, Aidan

Ding, Zishuo

Even-Mendoza, Karine

Kelly, David

Menendez, Hector D.

Moussa, Rebecca

Sarro, Federica

AI-based image generation is bounded by system parameters and the way users define prompts. Both prompt engineering and AI tuning configuration are current open research challenges and they require a significant amount of manual effort to generate good quality images. We tackle this problem by applying evolutionary computation to Stable Diffusion, tuning both prompts and model parameters simultaneously. We guide our search process by using Yolo. Our experiments show that our system, dubbed StableYolo, significantly improves image quality (52% on average compared to the baseline), helps identify relevant words for prompts, reduces the number of GPU inference steps per image (from 100 to 45 on average), and keeps the length of the prompt short (≈ 7 keywords)

King's Research Portal

                       King’s Research Portal  Document VersionPeer reviewed versionLink to publication record in King's Research PortalCitation for published version (APA):Berger, H., Dakhama, A., Ding, Z., Even-Mendoza, K., Kelly, D., Menendez, H. D., Moussa, R., & Sarro, F.(2023). StableYolo: Optimizing Image Generation for Large Language Models. In Symposium on Search-BasedSoftware Engineering (SSBSE) 2023 Springer.Citing this paperPlease note that where the full-text provided on King's Research Portal is the Author Accepted Manuscript or Post-Print version this maydiffer from the final Published version. If citing, it is advised that you check and use the publisher's definitive version for pagination,volume/issue, and date of publication details. And where the final published version is provided on the Research Portal, if citing you areagain advised to check the publisher's website for any subsequent corrections.General rightsCopyright and moral rights for the publications made accessible in the Research Portal are retained by the authors and/or other copyrightowners and it is a condition of accessing publications that users recognize and abide by the legal requirements associated with these rights.•Users may download and print one copy of any publication from the Research Portal for the purpose of private study or research.•You may not further distribute the material or use it for any profit-making activity or commercial gain•You may freely distribute the URL identifying the publication in the Research PortalTake down policyIf you believe that this document breaches copyright please contact librarypure@kcl.ac.uk providing details, and we will remove access tothe work immediately and investigate your claim.Download date: 10. May. 2024StableYolo: Optimizing Image Generation forLarge Language ModelsHarel Berger3[0000−0001−6035−5127], Aidan Dakhama1[0009−0002−7318−7964],Zishuo Ding4[0000−0002−0803−5609], Karine Even-Mendoza1[0000−0002−3099−1189],David Kelly1[0000−0002−5368−6769], Hector Menendez1[0000−0002−6314−3725],Rebecca Moussa2[0000−0001−9123−6008], and Federica Sarro2[0000−0002−9146−442X]1 King’s College London {aidan.dakhama,karine.even mendoza,david.a.kelly,hector.menendez}@kcl.ac.uk2 University College London {r.moussa,f.sarro}@ucl.ac.uk3 Georgetown University bergerar0@gmail.com4 University of Waterloo zishuo.ding@uwaterloo.caAbstract. AI-based image generation is bounded by system parametersand the way users define prompts. Both prompt engineering and AI tun-ing configuration are current open research challenges and they require asignificant amount of manual effort to generate good quality images. Wetackle this problem by applying evolutionary computation to Stable Dif-fusion, tuning both prompts and model parameters simultaneously. Weguide our search process by using Yolo. Our experiments show that oursystem, dubbed StableYolo, significantly improves image quality (52%on average compared to the baseline), helps identify relevant words forprompts, reduces the number of GPU inference steps per image (from100 to 45 on average), and keeps the length of the prompt short (≈ 7keywords).Keywords: LLMS · SBSE · Image Generation · Stable Diffusion · Yolo1 IntroductionGenerative AI [4] has proven to be one of the advances of the decade, disruptingthe way AI contributes to Arts. Large Language Models (LLMs), in combina-tion with generative models, provide interfaces that greatly scale up contentgeneration. The training of these systems normally consists of a massive train-ing dataset with associated descriptions. The correct description –or prompt–would create the desired outcome almost exactly. However, the level of vaguenessand ambiguity in the description can lead to unexpected results or errors.Image generation is a good example of this specific problem. When gener-ating images, generative models like DALL-E, Midjourney or Stable Diffusionrecommend variations to the prompt to guarantee that the quality of the gener-ated image is closer to expectations [9]. For example, if the user aims to create aphotorealistic image, it is recommended to include words like “professional pho-toshoot, 8k resolution, photorealistic masterpiece” or “natural lighting”. However,2 H. Berger et al.(a) Elephant, before and after (b) Zebra, before and afterFig. 1: Before-and-after snapshots of optimization with StableYolo of two images:an elephant and a zebra.it is not clear which words obtain the best results. Parameters of the algorithm,such as the number of steps used to generate an image, the initial random seedor the classifier-free guidance scale, significantly affect the outcome, as well. Thegoal is to create images under the specific condition of photo-realistic images.To this end, we propose a framework, StableYolo, which aims to improvethe quality of automatic image generation for photo-realistic images by leverag-ing the power of multi-objective search coupled with vision. The search processaims to improve different parameters of an image generation system (i.e., Sta-ble Diffusion) according to the outcome of the chosen visualization system (i.e.,Yolo). In this work, we use Stable Diffusion [2] and Yolo, respectively, but theframework can be extended to work with other systems. Our results show thatStableYolo significantly improves the quality of the generated images and helpsdirect prompts effectively across various contexts (Fig. 1). StableYolo providesa 52% increase in quality compared to the baseline results.Contributions The main contributions of this work are:(1) Creating the first system combining search and vision for image generation;(2) Testing this system on 42 different objects that can be obtained with imagegeneration and detected by our vision system; and(3) Obtaining significant improvements and prompt recommendations for thegeneration process of photo-realistic images.2 Proposed ApproachGiven a user-provided prompt of a photo-realistic image to a generative AI sys-tem, StableYolo attempts to improve image quality by finding an optimal com-bination of both, the prompt and the model’s parameters. It uses a search-basedprocess to identify an optimal combination, using Yolo’s confidence estimates asthe fitness function.StableYolo uses a Genetic Algorithm (GA) for the search process, where eachindividual is a dictionary object with values taken from StableDiffusion’s docu-mentation [11]. These include: (1) Number of iterations: the number of iterationsneeded for the AI to go through the image (range [1,100]); (2) Classifier-freeguidance scale (CFG): a parameter that controls the prompt’s influence on theStableYolo: Optimizing Image Generation for Large Language Models 3resulting image (range [1,20]); (3) Positive prompt : enables the prompt to de-scribe the images and their details; (4) Negative prompt : a sequence of keywordsto be avoided during the image generation process; (5) Seed : the generationseed for randomization; and (6) Guidance rescale: prevents over-exposure by re-scaling the guidance factor (range [0,1]). Positive prompt: In this work, we passto the prompt, words that aim to produce more realistic images, according todifferent images extracted from the Photo-realistic Prompts documentation [10];e.g., ’photograph’, ’digital’, ’color’, ’Ultra Real’ and ’award winning’. Negativeprompt: We select references that might reduce the realistic component of theimage; e.g., ’illustration’, ’painting’, ’drawing’ and ’art’.Utilizing GA, StableYolo optimizes both the LLM’s parameters and theprompts based on the encoding of each potential individual. We start with apopulation of a set of individuals initialized with randomly selected parameters.Next, we evaluate the individuals using the fitness function. The fitter the indi-viduals, the higher the chance they have of being chosen during the TournamentSelection process, after which the pair of individuals would undergo the crossoverand mutation operators.The crossover creates a pair of new offsprings by exchanging the attributesamong the selected parents. After, the mutation operator (if applied) changesthe parameters of an offpsring by uniformly assigning them new values within agiven range. This results in the creation of a new population after each iterationof the algorithm. A stopping criterion is required to end the algorithm which isusually indicated by a number of generations specified by the user.StableYolo aims to maximize its objective function through the followingsteps: (i) StableYolo creates positive and negative prompts and sets the parame-ters of Stable Diffusion to create four images per prompt; (ii) Yolo evaluates theimages by identifying objects within it. These objects have an identification con-fidence that StableYolo uses to evaluate image quality; and (iii) StableYolo thencalculates the fitness as the average quality of individual objects and images.This results in a different search objective based on the quality of each object.To avoid a multi-dimensional Pareto front, whose dimensions are unknown, thefitness calculates the object’s quality average obtained with Yolo.3 Experimental Setup and ResultsIn this section, we describe our research questions, experimental setup and re-sults.RQ We aim to answer the following research questions:- RQ1 To what extent can we improve the quality of an image generationsystem?- RQ2 What is the average prompt length to achieve maximum quality?- RQ3 What is the keyword frequency associated with quality images?In order to provide a wide set of elements to evaluate StableYolo, we used42 different categories of objects, animals and people that Yolo can recognize,4 H. Berger et al.airplanebananabearbedbicyclebirdboatbookbottlebrocollibuscakecarcarrotchaircouchcupdogdonutelephantforkfrisbeegiraffekitemicrowavemotorcycleovenpersonpizzarefrigeratorsandwichsinksnowboardsuitcase tietoothbrushtraintruck TVumbrellavasezebra0.00.20.40.60.81.0TopicQuality ScoreInitial QualityFinal Quality(a) detailed score per topic with and without improvementsFinal QualityInitial Quality0.00.20.40.60.81.0Quality Score(b) col. min-maxSetsFig. 2: RQ1: Quality Score Comparison: StableYolo (orange) vs. Non-Optimized(blue) Image Generation. StableYolo always produces better images on ourdataset.(e.g., person, elephant, zebra, dog, boat, and TV among others). For RQ2, weconsider a 1− 2 word prompt as small, 3− 6 words as medium and 7+ words aslarge.Experimental Setup For the GA, the settings were as follows; the populationsize was 25, the number of generations was set to 50, the mutation rate andcrossover were both set to 0.2, and the tournament selection value was set to5. Even given the small population size, we find the algorithm produces goodresults (Fig. 2a). The experiments were repeated 4 times. All experiments wereconducted on a workstation running Ubuntu 20.04 LTS with 36 CPU cores, 256GB RAM and an NVIDIA Titan V GPU with 12 GB memory.Results Fig. 2 shows the results based on Yolo’s quality assessment, rankedwith a score from 0 to 1, where 1 is best. Fig. 2a shows the quality score withand without improvements, grouped by topic. The orange line represents thequality score per topic after applying StableYolo, while the blue line denotesthe baseline before improvement (e.g., the topic baseline score is 0.558). Theblue and orange dotted lines represent the mean quality score before and afterimprovement, respectively. Fig. 2b presents the quality score across all topics,where the orange boxplot shows the results with improvements, and the blue oneshows them with no improvements. Fig. 2a shows improvement on all 42 prompts,with the orange line scores always being above the blue line scores. The initialprompt quality average is 0.567 with a standard deviation (SD) of 0.096, whilethe average score after optimization is 0.854 with an SD of 0.0836. This gives anaverage improvement of 52% in the quality scores after optimization. As the finalquality scores fail the normality test, we applied the Wilcoxon test under the nullhypothesis that both results follow the same distribution. The resulting p-valuefor the test is 4.55 · 10−13, clearly rejecting the null hypothesis, and showing astatistically significant difference between both sets of results. The optimized setdemonstrates a significant increase in the quality of the generated images.Parameters and Prompts The average optimal value for the guidance scale is9.65±3.99 in the range [0,20] while the rescale value is 0.56±0.29 on average. TheStableYolo: Optimizing Image Generation for Large Language Models 50 5 10 15 20 25 30 35 40 45photographdigitalfilm grainKodak portra 800Depth of field 100mmblended visualsUltra Realcoloroverlapping compositionsaward winningtrending on artstationillustrationpaintingsketcherrordrawingartcroppedlowresworst qualityWord Counts for Positive and Negative PromptsPositiveNegativeFig. 3: RQ3: Positive and negative prompt terms counting for the best individ-uals.number of inference steps to 45.3 ± 30.29 on average, in the range [0,100]. Thepositive prompt length on average is 6.5±1.66 and the negative one is 4.45±1.09.At the end of the optimization process, the most relevant keywords were foundto be ’photograph’ and ’Depth of field 100mm’ for the positive prompt, and’illustration’ and ’drawing’ for the negative one (Fig. 3). The memory cost tothe GPU was 9 GB, with each prompt optimization taking approximately 2hours to execute. On average, the GA takes 45.05±10.01 iterations to converge.StableYolo can scale through multiple GPUs linearly for both the generationand the visualization process.4 Related WorkGenerative AI systems have been used to perform the task of image generationfor many years [1,5,7,8]. Progress in this domain has resulted in incremental ad-justments or entirely novel architectures, including but not limited to generatorregularization [8], additional memory gate incorporation [12], use of dynamicthresholding [9], and fusion of neural architecture search with generative adver-sarial networks [6]. To the best of our knowledge, our work is the first to leveragethe power of EAs to improve prompts and model parameters simultaneously. Acomprehensive survey on other uses of LLMs can be found in Fan et al.’s work [3].5 ConclusionsWe introduce StableYolo, an innovative framework that enhances the fidelity ofphoto-realistic image generation by means of an iterative optimization proce-dure. StableYolo combines the text-to-image approach of Stable Diffusion v2 forimage generation making use of the image-to-text mechanism of Yolo v8 for cap-tion generation. An iterative process based on multi-objective search is adopted,6 H. Berger et al.utilizing affirmative and opposing prompts until convergence is attained. Oppor-tunities for further enhancement persist, most notably the current iteration ofStableYolo is constrained to classes identified by Yolo. Widening the spectrumof recognizable classes may enhance the diversity and analytical scope of gen-erated images. Nonetheless, StableYolo exhibits a remarkable enhancement inthe quality of image generation. In terms of generality, the StableYolo can easilybe extended to other image generation models like DALL-E or Midjourney anddifferent visualization systems apart from Yolo, for instance captioning systems,this will allow more objects and contexts to be addressed.Availability StableYolo is at https://github.com/SOLAR-group/StableYolo.Acknowledgments Authors are listed in alphabetical order. This work wassupported by the ERC grant no. 741278 (EPIC), the UKRI TAS Hub grant no.EP/V00784X/1 and EP/V026801/2.References1. Avrahami, O., Lischinski, D., Fried, O.: Blended diffusion for text-driven editingof natural images. In: IEEE/CVF CVPR. pp. 18208–18218 (2022)2. Chefer, H., Alaluf, Y., Vinker, Y., Wolf, L., Cohen-Or, D.: Attend-and-excite:Attention-based semantic guidance for text-to-image diffusion models. ACM Trans-actions on Graphics (TOG) 42(4), 1–10 (2023)3. Fan, A., Gokkaya, B., Harman, M., Lyubarskiy, M., Sengupta, S., Yoo, S., Zhang,J.M.: Large language models for software engineering: Survey and open problems(2023)4. Jo, A.: The promise and peril of generative ai. Nature 614(1), 214–216 (2023)5. Kang, M., Zhu, J.Y., Zhang, R., Park, J., Shechtman, E., Paris, S., Park, T.: Scalingup gans for text-to-image synthesis. In: IEEE/CVF CVPR. pp. 10124–10134 (2023)6. Li, W., Wen, S., Shi, K., Yang, Y., Huang, T.: Neural architecture search with alightweight transformer for text-to-image synthesis. IEEE Transactions on NetworkScience and Engineering 9(3), 1567–1576 (2022)7. Li, W., Zhang, P., Zhang, L., Huang, Q., He, X., Lyu, S., Gao, J.: Object-driventext-to-image synthesis via adversarial training. In: IEEE/CVF CVPR. pp. 12174–12182 (2019)8. Mao, Q., Lee, H.Y., Tseng, H.Y., Ma, S., Yang, M.H.: Mode seeking generativeadversarial networks for diverse image synthesis. In: IEEE/CVF CVPR. pp. 1429–1437 (2019)9. Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E.L., Ghasemipour,K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., et al.: Photorealistic text-to-image diffusion models with deep language understanding. Advances in NeuralInformation Processing Systems 35, 36479–36494 (2022)10. Stable Diffusion Photorealistic Prompts: https://prompthero.com/stable-diffusion-photorealistic-prompts11. The Stable Diffusion documentation: https://huggingface.co/docs/diffusers/api/pipelines/stable_diffusion/text2img12. Zhu, M., Pan, P., Chen, W., Yang, Y.: Dm-gan: Dynamic memory generative adver-sarial networks for text-to-image synthesis. In: IEEE/CVF CVPR. pp. 5802–5810(2019)

English

StableYolo: Optimizing Image Generation for Large Language Models

StableYolo: Optimizing Image Generation for Large Language Models

Abstract

Similar works

Full text

Available Versions

King's Research Portal