You can fine-tune LLMs to be more creative without sacrificing quality by teaching them to use special control tokens, and this creative capability actually helps downstream tasks like reinforcement learning.
This paper introduces CreativeInstruct, a method to teach language models to generate more creative and diverse outputs while maintaining quality. The approach uses special tokens to guide models toward creative generation, includes a new diversity metric based on graph structure, and shows improvements in story generation and math reasoning tasks.