lossless to lossless compression

I think you have confused lossless with lossy compression.

Lossless formats (FLAC and others) include all the information which was in the original CD audio track; they just write it in a shortened form. If you've encoded into a lossless format, you've "lost" nothing. It's like putting files into a ZIP archive - totally reversible. Lossless to lossless compression affects only the file's size, not its quality.

Lossy formats, like MP3 and AAC, can make smaller files, because they throw away some of the information from the CD. You can never recover the original CD-quality track. It's like writing a summary of a longer document; the MP3 or AAC encoder summarises the original file, preserving the most 'important' bits.

Converting from 320k MP3 to 128k MP4 - well, your end product would be worse than if you were to go back to the original CD file and rip straight into 128k MP4. How much worse is a matter for your own personal judgement, for sure. Why not try a few and see whether you like the output?