Thank you so much for the nice comments and tips on how to make it work. It gave me the energy to go back in and work on it. Without your support, it might have gone to the refashion stockpile and waited another couple of years.
It took me a while to figure out what the problem was and what to do about it. But you can actually see what going on in this before picture. As I am pinching, the side seam is curving towards the back, which meant the excess fabric was in the front.
If you need to know, here is what I did : I opened up the front skirt to bodice stitch, put the dress inside in, pinned what needed to be taken in, which was 3cm on both sides.
I traced the stitching lines, sewed them, ran a gathering stitch on the skirt and sewed it back in. This dress is now a bloody mess inside but this is no change from it's original state so I can live with that. It'll need to be hand washed to prevent damages but, this is the extend of my patience with this project and this'll be it.
Oups, I lied! I did change the buttons, my favorite haberdasher makes such pretty covered buttons.