Last week, enthusiasts trying out the new Flux AI image synthesis model discovered that it was surprisingly good at rendering facsimiles of custom-trained typefaces. While much more efficient ways to display computer fonts have existed for decades, this new technique is useful for AI image enthusiasts. Because Flux can render accurate representations of text, users can now insert words rendered in custom fonts directly into their AI image generation.
Creating AI-replicated fonts isn’t big news in itself, since the technology has existed since the 1980s (and in research since the 1970s) to accurately create smooth, computer-rendered fonts with custom shapes. But new techniques could mean that a particular font could appear in AI-generated images, like a photorealistic restaurant chalkboard menu or a printed business card held by a cyborg fox.
Shortly after the emergence of mainstream AI image synthesis models like Stable Diffusion in 2022, some people began to wonder, “How can I inject my own product, clothing item, character, or style into an AI-generated image?” One answer emerged in the form of LoRA (Low Rank Adaptation), a technique discovered in 2021 that allows users to extend the knowledge of AI-based models with custom-trained modular add-ons.
Cyberpunk 2077 LoRA example rendered with Flux dev.
Cyberpunk 2077 LoRA example rendered with Flux dev.
Cyberpunk 2077 LoRA example rendered with Flux dev.
Cyberpunk 2077 LoRA example rendered with Flux dev.
These modules, called LoRAs, allow image synthesis models to create new concepts that were not originally present (or well represented) in the training data of the underlying model. In practice, image synthesis enthusiasts use them to render their own unique style (e.g., all of chalk art) or subject matter (e.g., a detailed image of Spider-Man). Each LoRA must be specially trained with examples provided by the user.
Until Flux, most AI image generators weren’t very good at rendering accurate text in a scene. Ask Stable Diffusion 1.5 to render a sign that says “Cheese” and it will return gibberish. OpenAI’s DALL-E 3, released last year, was the first mainstream model that handled text reasonably well. Flux still makes mistakes with words and characters. But it is the most capable AI model I’ve seen to render “in-world text” (as you might call it).
Flux is an open model that can be downloaded and tweaked, so this past month has been the first time that LoRA training of a typeface has made sense. It was recently discovered by AI enthusiast Vadim Fedenko (who did not respond to an interview request at the time of writing). “I’m really impressed with the results,” Fedenko wrote in a Reddit post. “Flux recognizes what characters look like in certain styles and fonts, allowing us to train Loras on specific fonts, typefaces, etc. We plan to do more training on it in the near future.”
The first Flux typeface, LoRA, a Y2K example.
Y2K LoRA example.
Y2K LoRA example.
For his first experiment, Fedenko chose a dynamic, “Y2K”-style font, reminiscent of fonts popular in the late 1990s and early 2000s, and published the resulting model on the Civitai platform on August 20. Two days later, a Civitai user by the name of “AggravatingScree7189” posted a second typeface, LoRA, which recreated a font similar to one featured in the Cyberpunk 2077 video game.
“I had no idea this was possible because the previous text was awful,” wrote Reddit user eggs-benedryl in response to Fedenko’s post about the Y2K font. “I didn’t realize the Y2K journal was fake until I zoomed in,” another Reddit user wrote.
Is that too much?
Admittedly, using a well-trained image synthesis neural network to render old fonts on a plain background may be overkill, and you probably wouldn’t want to use it as a replacement for Adobe Illustrator when designing documents.
“This looks good, but reinventing the concept of fonts as a 300MB LoRA is a bit weird,” wrote one Reddit commenter in a thread about Cyberpunk 2077 fonts.
Generative AI is often criticized for its environmental impact, a legitimate concern in large cloud data centers. But Flux found that it could insert these fonts into an AI-generated scene in quantized (reduced size) form while running them locally on an RTX 3060 (the full development model can run on an RTX 3090), with a similar power consumption to playing a video game on the same PC. The same can be said about the creation of LoRA: the creators of the Cyberpunk 2077 font trained LoRA for three hours on a 3090 GPU.
The use of AI image generators also raises ethical questions, such as how they are trained on data collected without the consent of content owners. While the technology has been controversial among some artists, a large community of people use it every day and share their results online through social media platforms such as Reddit, which has led to new applications of the technology like this one.
At the time of writing, there are only two custom Flux typefaces, LoRA, but we are already hearing plans to create more as of writing. Although it is still early days, the technology behind creating the LoRA typefaces could become the basis if AI image synthesis becomes more widely adopted in the future. Adobe, which has its own image synthesis model, is probably taking notice.