More weight or less weight?

So when I do upper body work I tend to feel it more the next day in my lower body and not so much in my upper body. I feel like I'm lifting a good weight for me and that my form isn't bad (I video tape myself to make sure).

But if I'm not sore the next day in my upper body should I increase the weight? Will that make me extra sore in my lower body?