Ex code for CoarseTransformer and FineTransformer

[raghavpatnecha]

I have been able to successfully train SemanticTransformerTrainer. But getting error with later two.
coarse_transformer = CoarseTransformer(
codebook_size=wav2vec.codebook_size,
num_coarse_quantizers = 8,
num_semantic_tokens = 1000,
dim=1024,
depth=6,
audio_text_condition=True # this must be set to True (same for SemanticTransformer and FineTransformer)
).cuda()

trainer = CoarseTransformerTrainer(
transformer=coarse_transformer,
wav2vec=wav2vec,
audio_conditioner=quantizer, # pass in the MulanEmbedQuantizer instance above
folder='/content/music_data',
soundstream=soundstream, #where to get this from
batch_size=1,
data_max_length=320 * 32,
num_train_steps=1
)`

[lucidrains]

@raghavpatnecha what error do you see?

[raghavpatnecha]

soundstream = SoundStream.init_and_load_from('/content/results/semantic.transformer.0.pt')

trainer = CoarseTransformerTrainer(
transformer = coarse_transformer,
soundstream = soundstream,
wav2vec = wav2vec,
folder = '/content/music_data',
batch_size = 1,
data_max_length = 320 * 32,
num_train_steps = 1
)

trainer.train()

I am using soundstream as mentioned in audiolm and getting below error

[raghavpatnecha]

Do I need to train soundstream separately ?

[lucidrains]

@ahh yea, you are trying to load the semantic transformer checkpoint into soundstream

and yea, soundstream needs to be trained first. ideally there is a coordinated effort to train and open source a few foundation soundstream models