🎵 Oh GPT, Oh GPT, Please Generate Some Carols 🎵

One of the best parts of the Christmas season is people no longer judge you for listening to All I Want for Christmas is You on repeat! There are some great Christmas tunes out there, but sometimes you just want something new. Your only options are to wait for Michael Bublé to drop a new Christmas album, pen some songs yourself, or train a neural network to write Christmas carols for you.

It’s Christmastime now, so I don’t have time to wait for a new album or to write my own songs! So the only option was to train GPT-2 on Christmas carol lyrics and have it generate some new ones for me. Which I did. And you can generate lyrics too because I integrated it into a web app!

https://christmas-carol-generator.anvil.app

How I Built It

I used the Hugging Face Transformers library and their example scripts to fine-tune GPT-2 and generate Christmas carols. I found a dataset of christmas songs here.

After re-training GPT-2 on this dataset, I made some minor changes to Hugging Face’s run_generation.py script so that I could use the Anvil Uplink to connect to my app.

First, I added the argument prompt_text to the main() function so that starting lyrics could be passed to the model from the web app. I modified the script slightly to use this prompt text. Next, I changed some of the defaults of the argparse arguments so that the script defaulted to using GPT-2 and the directory containing my fine-tuned Christmas model. In order to vary the model’s output, I randomized the seed used to initialize the model.

Finally, I connected my local script to my Anvil app using the Uplink. I opened a connection to the Uplink from my local script and made my local main() function callable in Anvil. I also modified the main() function to return the generated lyrics so I could display these on a Label in my web app.

#connect to local script through Uplink
import anvil.server
anvil.server.connect("<uplink-key>")

...

def set_seed(args):
#randomize seed when script is run 
   seed = random.randint(1, 100)
   np.random.seed(seed)
   torch.manual_seed(seed)
   if args.n_gpu > 0:
       torch.cuda.manual_seed_all(seed)

...

#make function callable in web app
#make gpt-2 and fine-tuned model the default
@anvil.server.callable
def main(prompt_text):
   parser = argparse.ArgumentParser()
   parser.add_argument(
       "--model_type",
       default='gpt2',
       type=str)
   parser.add_argument(
       "--model_name_or_path",
       default='./tuned_model',
       type=str)

    ...

   return total_sequence

#keep connection to uplink open
anvil.server.wait_forever()

I then designed a wintery background and picked a Christmas-y font to make my app extra festive! Now all that’s missing is music…

If you want to investigate the app, you can clone it with the link below:

P.S. Don’t expect GPT-2 to generate good lyrics. The dataset I used was small and unannotated, so it’s not going to be enough for a neural network to pick up on syllables or rhyming schemes. Christmas songs are also mainly either religious, about Santa Claus, or a love song, and these disparate themes really seem to confuse GPT-2.

P.P.S. I also fine-tuned GPT-2 for Halloween so that it would generate stories in the style of Edgar Allan Poe. You can read about it and generate your own spooky ghost stories here!