# Taking ImgFlip's 'This Meme Does Not Exist' to the Next Level with a LLM Stefan Ojanen, Genesis Cloud | MLOps Community | 14:49 Source: https://www.youtube.com/watch?v=ejaee9IZw3E Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks. Page: https://mlopstalks.com/talks/taking-imgflips-this-meme-does-not-exist-to-the-next-level-with-a-llm Published: 2023-07-03 Tags: cost, fine-tuning, model-serving, open-models ## TL;DR - A dataset of 40 million user-created memes let Genesis Cloud fine-tune a commercially usable language model for meme generation. - StableLM produced the best results in their comparison, while earlier models often struggled with meme structure and output format. - A self-hosted model could cost much less than the OpenAI API at high request volumes, while giving the team more control over data and availability. ## Summary Stefan Ojanen describes a Genesis Cloud project that uses a large language model to generate captions for ImgFlip meme templates. The team worked with 650,000 templates and 40 million user-created memes, then narrowed the initial target to 256 templates that covered 62% of created memes. GPT-3.5 showed some understanding of meme context but also hallucinated and missed template structure. After trying several open models, the team found that StableLM worked best for the task. They used template-specific prompts and fine-tuned the model with Lit-Parrot, LoRA, and an adapter. The proof of concept trained on four GPUs in 21 minutes. For a possible production service, Stefan estimates a setup that can scale to about 40 GPUs, with Kubernetes handling scaling and Triton serving the model. He compares this approach with GPT-4, citing control, reliability, data ownership, and cost as reasons to run the model themselves. ## Key ideas ### Meme generation depends on understanding each template's structure [01:31](https://www.youtube.com/watch?v=ejaee9IZw3E&t=91s) Stefan describes memes as existing material with small variations and says humor remains poorly understood. GPT-3.5 could produce a reasonable Philosoraptor caption, but it also hallucinated and failed to understand the template structure. The comparison with a human-generated example shows why generic text generation is insufficient. A Philosoraptor meme needs a silly but smart question, such as the joke about the opposite of progress being congress. The project therefore treats meme context and format as part of the generation problem. ### The ImgFlip dataset combines templates with real user-created captions [04:55](https://www.youtube.com/watch?v=ejaee9IZw3E&t=295s) ImgFlip provided a dataset containing 650,000 meme templates and 40 million individually created memes. Stefan notes that many templates are nearly duplicates, so the number of genuinely unique templates is much smaller. The team wanted to extend the existing service to more templates and chose 256 as its first target. Those templates covered 62% of all memes created on the service. The dataset gave the team examples of how users actually write captions instead of relying only on synthetic prompts. ### StableLM was the best fit among the models they tried [06:01](https://www.youtube.com/watch?v=ejaee9IZw3E&t=361s) The team first tried a BERT-based approach and moved on quickly after getting poor results. They also tested Llama-related models, RedPajama, and Pythia. Some outputs were funny, but the models often struggled with the required format or produced nonsense. StableLM worked best in their tests. Stefan says it was still behind GPT-4, but close enough to make the project worthwhile and potentially improve further. Commercial licensing also mattered because one Llama version they considered did not permit commercial use. ### Template-specific prompting and lightweight fine-tuning produced the proof of concept [07:01](https://www.youtube.com/watch?v=ejaee9IZw3E&t=421s) The chosen approach gives the model an overall instruction describing the meme format, then supplies the specific context for each template. Stefan says this worked better when the team created that prompt structure separately for every template. They used Lit-Parrot with LoRA and an adapter for fine-tuning. On a preliminary dataset, three epochs were enough for the model to converge. Training took 21 minutes on four GPUs, and the team was able to keep those GPUs fully occupied rather than wasting resources. ### The early evaluation was small and subjective [08:01](https://www.youtube.com/watch?v=ejaee9IZw3E&t=481s) For the initial results, the team generated seven iterations for each of three templates using the models under consideration. Stefan graded the outputs himself and judged 76% funny. He presents examples such as 'Work harder drink' and 'Vitamin taco laugh', while also showing captions that are grim or awkward. He is clear that the sample was small and that the evaluation reflected his own judgment. The examples demonstrate that the model can produce usable jokes, but they do not establish broad quality across the full dataset. ### A production service would need capacity planning around request volume [09:29](https://www.youtube.com/watch?v=ejaee9IZw3E&t=569s) The planned system would fine-tune the model for 256 templates and expose it through an API that ImgFlip could use. Based on an estimate of six tokens per second on an NVIDIA 3090, about ten tokens per request, and roughly five million monthly requests, Stefan calculates a baseline of about 40 GPUs without seasonality. He expects peak load to require scaling up to that level. The proposed architecture uses Kubernetes for scaling, monitoring around the service, and Triton to serve the model. ### Self-hosting offers control and lower cost than relying entirely on GPT-4 [10:44](https://www.youtube.com/watch?v=ejaee9IZw3E&t=644s) Stefan gives several reasons not to build the service solely on GPT-4. The OpenAI API is a dependency for the application, the team does not control its data path or model, and the cost can become high at scale. Genesis Cloud's fine-tuning cost was about one US dollar in their setup. Generating 50 million tokens would cost roughly 300 to 500 US dollars per month depending on the purchasing model, while Stefan estimates about 4,500 US dollars through the OpenAI API. He still credits OpenAI's work and frames this as a case where running a smaller model can be practical. ### The unresolved engineering work is concurrency, latency, and service-level targets [11:47](https://www.youtube.com/watch?v=ejaee9IZw3E&t=707s) The proof of concept was working, but Stefan had not yet completed the production design. The team still needed to determine the minimum infrastructure that could meet its service-level agreement while controlling latency. He questions whether CPU utilization is a good enough signal and suggests CPU queue time as a better leading indicator. Handling a large number of concurrent requests was another open problem. A later version was expected to include the full dataset and the actual production challenges rather than only the issues anticipated in the prototype. ## Notable quotes - "The goal was to showcase how a content aware LLM can create quality memes." (05:39) - "Stable LLM, this is where we struck gold." (06:42) - "So using four GPUs it was only 21 minutes to fine tune it." (08:01) - "OpenAI API becomes a single point of failure for your AI application." (10:44) - "Humor is difficult, right?" (13:46) ## Tools & references mentioned - Genesis Cloud - ImgFlip - This Meme Does Not Exist - GPT-3.5 - GPT-4 - Dank learning - LSTM - Dylan - Llama - OpenLLaMA - RedPajama - Pythia - StableLM - Lit-Parrot - LoRA - Kubernetes - Triton - OpenAI - Demetrios Brinkmann ## Who should watch - You are building an LLM feature around a narrow content format and want to see how real user examples can shape prompting and fine-tuning. - You need to compare a hosted API with a self-hosted model on licensing, cost, data control, and operational complexity. - You are taking a model from a quick proof of concept toward a high-volume service and want concrete questions about capacity, latency, and concurrency. ## Related talks - [The Next Revolution in AI: LLMs and Beyond](https://mlopstalks.com/talks/the-next-revolution-in-ai-llms-and-beyond) (, 13:47) - [Making LLM Inference Affordable](https://mlopstalks.com/talks/making-llm-inference-affordable) (Daniel Campos, Snowflake, 32:07) - [LLMs For the Rest of Us](https://mlopstalks.com/talks/llms-for-the-rest-of-us) (Vikram Sreekanti, Aqueduct & Joseph Gonzalez, UC Berkeley and Aqueduct, 24:33) - [LLMs vs LMs in Production](https://mlopstalks.com/talks/llms-vs-lms-in-production) (Denys Linkov, Voiceflow, 24:44) - [Scalable Evaluation and Serving of Open Source LLMs](https://mlopstalks.com/talks/scalable-evaluation-and-serving-of-open-source-llms) (Waleed Kadous, Anyscale, 34:57)