I lose more weight when I don't care

About four weeks ago I went on vacation and didn't exercise or eat healthy because I just wanted to enjoy my time there without stressing about whether my next meal would be healthy or finding time to workout. The week long trip was also followed by about a week of jet lag which meant I accidentally skipped a lot of meals because I slept through them (I napped a lot). I also didn't work out during those days because I had no energy whatsoever.

Overall I had lost about 3-4 pounds. It's been two weeks now and I'm back eating regularly and I found that I was gaining weight again even though I'm eating smaller portions and healthier food than on vacation and started working out again.

Can someone explain what I need to do to start losing weight again?