WACV 2025
Transformer based methods have enabled users to create, modify and comprehend text and image data. Recently proposed Large Reconstruction Models (LRMs) further extend this by providing the ability to generate high-quality 3D models with the help of a single object image. These models, however, lack the ability to manipulate or edit the finer details, such as adding standard design patterns or changing the colour and reflectance of the generated objects, thus lacking fine-grained control that may be very helpful in domains such as augmented reality, animation and gaming. Naively training LRMs for this purpose would require generating precisely edited images and 3D object pairs, which is computationally expensive. In this paper, we propose Instructive3D, a novel LRM based model that integrates generation and fine-grained editing, through user text prompts, of 3D objects into a single model. We accomplish this by adding an adapter that performs a diffusion process conditioned on a text prompt specifying edits in the triplane latent space representation of 3D object models. Our method does not require the generation of edited 3D objects. Additionally, Instructive3D allows us to perform geometrically consistent modifications as the edits through user-defined text prompts are applied to the triplane latent representation thus enhancing the versatility and precision of 3D objects generated. We compare the objects generated by Instructive3D and a baseline that first generates the 3D object meshes using a standard LRM model and then edits these 3D objects using text prompts when images are provided from the Objaverse LVIS dataset. We find tha Instructive3D produces qualitatively superior 3D objects with the properties specified by the edit prompts.
Architecture of our method Instructive3D. The triplane is first generated by the LRM encoder (in this case Real3D), then each plane of the triplane is then separated, normalized between [-1,1] and processed through its dedicated encoder, trained specifically for the corresponding plane. The resulting latent planes have their channels concatenated and passed through a conditional UNet model for denoising, in conjunction with a text embedding obtained from a CLIP transformer based on the input text prompt. The denoised output is then separated back into the three planes, which are passed through their respective decoders. Finally, the planes are stacked together to form the conditioned triplane, reflecting the user-specified text-based modifications. Finally, the conditioned triplane is passed through the decoder of LRM model to obtain textured mesh and rendered video.