Generative AI for Design and Architecture: explorations and evaluations.

Carlo Turri iNEST Spoke 3 - RT4 "Artificial Intelligence and Data Science", Task 4.5 “Interaction, Visual and Digital Storytelling for Design, Service and Sharing” University IUAV of Venezia TECHNICAL NOTE Generative AI for Design and Architecture: explorations and evaluations. April - September 2024 THE CONTEXT: OPEN-SOURCE TECHNOLOGIES A fundamental premise of this research is that arriving at definitive conclusions is not feasible, as changes in this field are incessant and, particularly within the open-source domain, almost entirely unpredictable. Although the investigation occasionally extends to platforms of various kinds, the research scope has been delimited by two factors: the ability to use these technologies locally (through computers capable of running the models) and the focus on solutions developed within the open-source ecosystem. These two aspects are naturally correlated: the ability to manipulate or alter software is possible only when access to it is physically independent of proprietary digital infrastructures, and when those working on it are not subject to any restrictions. Most importantly, it should be emphasized that the primary drivers of innovation in this field stem from scientific research, and thus, the validity of these advancements must adhere to on the premise of publicly accessible data, as well as the principles of replicability and validation. Conversely, when technology is the product of private-sector investment, such intentional accessibility is usually precluded due to adhering to the most fundamental economic and strategic principles: once an innovation is achieved, its value must be maximized and, therefore, protected. Nevertheless, one of the most distinctive aspects of the open-source approach—along with the underlying philosophy—is its ability to activate mechanisms of collaboration and knowledge sharing. This often results in significant developments, both in terms of sophistication and widespread adoption, as isolated ideas or intentions sometimes yield groundbreaking outcomes. In summary, this research has focused on the open-source ecosystem for the following reasons: • The ability, within the limits of disciplinary expertise, to comprehend the underlying structure of the reference technological framework. • The dynamism with which it evolves and branches out. • The extensive variety that this complex landscape presents. • The opportunity to identify the most suitable tools and procedures for the disciplinary field of Design and Architecture. However, the open-source software landscape presents certain challenges, the most significant being that contextual information about the technologies in question is often fragmented, heterogeneous, and only partially organized. This makes access to these tools less immediate and their practical application more cumbersome. Compared to widely established digital tools, open-source technologies exhibit issues such as instability, compatibility challenges, and malfunctions. As a result, users must actively search for, test, and troubleshoot these inherent complications, which are common across the entire open-source ecosystem. This instability also manifests in terms of discontinuity and non-linearity of development: alongside continuous bursts of innovation, the open-source domain frequently generates obsolescence, stagnation, lack of resources, and project abandonment. KEY DEVELOPMENTS IN 2024 During this research period, several major technological advancements have shaped the field: • The consolidation of platforms such as Hugging Face (dedicated to sharing software components, usage guidelines, and development environments) and CivitAI (specialized in the distribution of models and dedicated modules). • The maturation of the ComfyUI graphic interface (released in 2023) and its establishment as a prominent tool, enabling rapid integration of the latest innovations while facilitating the widespread adoption of structured workflows invented for specific functions. • The release of Stable Diffusion 3 (by Stability AI) and its controversial, restrictive licensing terms, which shifted user interest toward alternative systems. • The refinement and diffusion of IP-adapter, an integrative model allowing image generation to be conditioned by visual references (image prompts) rather than solely by textual descriptions (prompts). Reference • The emergence and rapid adoption of FLUX.1 (by Black Forest Lab), which has become the new reference point for open-source developers, researchers, and user communities. • The recent introduction of CogVideoX in the domain of local video generation—despite significant inherent limitations affecting both the creative process and, more critically, model training. Reference OBJECTIVES By shifting the focus toward generative processes themselves, and thus toward the relationship between communication needs and what technology can offer to designers and architects, it becomes evident that these innovations are fundamentally transforming the way visual content is produced. This shift enormously amplifies the designer’s capabilities, both quantitatively and qualitatively. However, despite these remarkable advancements, certain limitations hinder the full and seamless integration of these technologies in already consolidated work-flows. These challenges are inherent to the functioning of generative models and generally manifest in the lack of control over the generative process compared to traditional digital design tools such as 3D modeling and rendering. In AI-based generation, the same input (whether a visual or textual prompt) does not consistently yield identical results—an element of randomness is intrinsic to the system’s operation. Moreover, slight variations in a few parameters can result in substantial differences between images (in terms of form, color, detail, and lighting). This limitation currently prevents designers from fully translating their vision and thought processes into a coherent set of related images. Consequently, it also restricts the ability to visualize an object or architectural structure in its complete and articulated form. Nevertheless, improving generative techniques to enhance user control and manipulation has become a key objective within the community. Accordingly, this research is rooted in this specific area of exploration. Beyond striving for optimal image quality and minimizing errors, the core question guiding this study is: which tools and techniques allow for the highest levels of coherence, precision, and versatility in generative image production? INVESTIGATIONS Having outlined the general technological landscape and the dynamics of the open-source context—along with its inherent volatility—this study has identified the most significant recent innovations and defined a specific research focus on tools and techniques that support the designer’s creative process. This provides a foundation for systematically reviewing the various investigative approaches undertaken in this research. Chart 1 Overview on the explored tools, working environments, models, and techniques. The exploration considers and compares multiple platforms, models, techniques both in the areas of image making and video (animation) creation. image gene ral framing of tec hnologies Easy Diffusion Easy DiffusionPikaLab promp ting img2imgimg2mesh img2vid vid2vid inpaintinganimatediffframe interlpolation textual inversion LORA dreambooth outpainting controlnet canny normal depth segmentation linea rt openpose mediapipe face (tile) conditional promp ting relightregional guidanceT2I adapter IP adapter Genie Meshy Haiper Stable Video Diffusion DoraAI (VizCom) Invoke AI ComfyUI ComfyUI (atimate diff ) BASIC TECHNIQUES ADVANCED TECHNIQUES TRAINING TECHNIQUES TOOLSCONTEXT video 3D web design From sketches to pictures The image2image based techniques have been largely employed since the advent of visual generative AI. Although the first results can be surprisingly impressive, the transformation of a simple sketches into a fully detailed and consistent image is not an easy task without employing a set of additional software called ControlNet (2023, Reference). Specifically, techniques based on edge detection (canny, HED, line art, etc.) can provide satisfactory results by ‘forcing’ the written prompt match the images to the given countours. Nevertheless, it is often necessary to remove errors by local refinements (inpainting). Interestingly, working progressively on a sketch (or retracing an original one step by step, as an alternative) can lead to a slower but more controlled result. This way the designer can progressively and iteratively check the quality for each generation. Figure 1 Progressive sketching allows a “by blocks” control of the key aspects of the image, interestingly driven by a rough envisioning of the most prominent visual features. While testing different model, we immediately noticed how the source material used to train or fine-tuning the model it is heavily impacting the visual result. As expected, models specialized in architectural imagery have been consistently trained on a basis of polished 3D renders and professional photographs of pristine and brand-new spaces: in fact, such models can easily replicate such look. Conversely, models trained for general photorealism are likely referencing a massive data set which comprises more diverse and casual photographic material, presumably taken mainly from social network. Such observation opens to interesting questions: can such unpolished imagery -and yet physically accuratebe useful to visualize architectural spaces beyond the current aesthetical conventions? Can this help to visualize how the built spaces will look under poor light conditions or when they will be aged? Figure 2 Comparison between two different AI models: architectural specialized model (Architecture Interior V8, on top row) versus a generic photorealistic one (Dreamshape 8 Turbo, on the bottom row). Although with subtle ye veryt visible inconsistencies, the latter one provides a minor artificial sensation. 3D as a common ground to aid the generation. Among the ControlNet technology there are few methods that rely on visually codified information: similarly to the previous examples, such auxiliary images provide further guidance to the written prompt, thus forcing the model to respect certain visual boundaries. We thoroughly experimented with the following ones: depth maps (a grayscale image which defines the proximity of the portrayed elements in relation to the point of view), normal maps (RGB images that provides the orientation of the portrayed elements in a tridamentional space), segmentation maps (in which a restricted colour palette is used to semantically define corresponding entities) . Along with the more conventional techniques (image denoise and the aforementioned contours-based ControlNet techniques), using such additional visual references has led to most reassuring results in terms of overall control and consistency. Such techniques have been tested in both interior design and urban planning type of imagery, both of which requires multiple and consistent visualizations: in such visual material, a road cannot be mistaken for a canal, and neither a window cannot be mistaken for a door. Such gross errors are quite diffused AI generative processes, hence the necessity to nudge the models to do not misinterpreting the visual provided information. In this regard, segmentation map-based approaches have proven to be quite reliable in generating similarity when with different style and lighting conditions are prompted; depth map-based techniques showed a better interpretation of the overall composition of the scene; although in a lesser way, normal maps may increase the control in light consistency (light direction and shadows). Figure 3 Semantic labelling of a segmentation map image. The most interesting aspect of such exploration has been linking the fabrication of such auxiliary images set to the same source, that is a fair simple 3D scene in which is possible to export from. Figure 4 From a 3D file, to conditioning images, to the generated ones. This way it is possible to change the point of view while keeping an acceptable level of consistency, as well as assigning different values to the colour maps. Such approach can therefore be applied to the other types of information that a 3D environment easily provides and establishing combination of how such additional imagery guide the generative process. Such approaches have been also tested in video generation, by processing image sequences. The result are interesting and surprisingly different from all the other animation forms, but such generative approach cannot still stand the precision and visual consistency produced from 3D rendering techniques. Figure 5. A ComfyUI workflow in with generation is steered via masks, depth, and HED lines. Nevertheless, the benefits of emplying such techniques can be helpful during early stage in the design process, in it is necessary to consider multiple nuances of the same idea/concept but there aren’t enough resources (time, costs, etc.) to undertake a complete 3D graphics workflow. Images in dialogue: IP-Adapters and visual prompting The strive to align users’ desired outcomes and the generative results has been significantly improved with the advent of the IP Adapter. In this case, this software components behave as auxiliary models, specialised into deconstructing the reference image and extracting and applying a range of visual information. It is a technique derived from image recognition studies, but it works in the ‘latent space’, which is an intermediary step between the semantics of images (subject, background, objects, etc.) and actual RGB values. In other terms, IP-Adapters tend to transfer visual features from a reference image(s), to the generated ones. Such approach is often used for face-replacement tasks, or stylization of an image in terms of colour scheme, overall look, and composition. This is also an extremely powerful solution to transfer textures, patterns, and other details that otherwise would result nearly impossible to be described via ordinary textual prompts. Figure 6 IP-Adapter technique used for affecting several part of the generative image (Source: InvokeAI). The most valuable trait of IP-Adapters is the capability to inject unique visual traits of a given subject even if it is not belonging to the main model parameters, thus bypassing the limitations of the model itself. In other terms, whereas a textual prompt-based generation lays on semantic correlations between the meaning of the words and the relative visual vocabulary, a visual prompt works following a straightforward principle of mimesis. We believe that such features is extremely important in bypassing the various models’ biases inherited from the data-sets the are based on, and, most importantly, such technology can help creators to achieve the uniqueness they are looking for. IP-Adapters have then been tested to fulfil the specific task (Material Exploratory integrated project) to portray objects looking being made of specific and niche materials. Figure 7 IP-Adapter technique combined with ControlNet and Regional Prompting. Figure 8 Testing results on IP-Adapters in reproducing the look of a material taken from sample pictures applied to specific object shapes. The same approach has been tested into texture generation, thus exploiting once more the capabilities to mimic the appearance directly from visual references. The resulting image texture can be then used as a base for creating PBR digital materials, to be later employed in traditional CG graphics application. From here on, models and materials can be exported in format glTF, and therefore ready to be shared online [see related documentation]. Figure 9 Creation of texture: from picture to AI generated texture, to PBR material to CG 3D software. References and sources Ramesh, A., Pavlov, M., Goh, G., Gray, S., & Misra, V. (2021). Zero-shot text-to-image generation. OpenAI. Retrieved from https://arxiv.org/abs/2102.12092 Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., & Mishkin, P. (2022). GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models. OpenAI. Retrieved from https://arxiv.org/abs/2112.10741 Wang, X., Zhang, Y., Li, T., & Liu, J. (2023). ControlNet: Adding conditional control to text-to-image diffusion models. arXiv preprint. Retrieved from https://arxiv.org/abs/2302.05543 Hugging Face Community. (2024). Hugging Face documentation & open-source AI research. Retrieved from https://huggingface.co Stability AI. (2024). Stable Diffusion 3: Advances and challenges in open-source image generation. Retrieved from https://stability.ai Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., & Chen, M. (2022). Hierarchical text-conditional image generation with CLIP latents. OpenAI. Retrieved from https://arxiv.org/abs/2204.06125