Scanline optimization (SO) is one of the fundamental refinement approaches in stereo matching. SO is incorporated to smooth final disparity estimations by using energy minimization over local patches. The main drawback of SO approaches is their constant energy penalties. The constant penalties may smooth the wrong disparity estimations as well as the correct estimations. In this paper, we propose an adaptive scanline optimization approach where the constant energy penalties are adaptively set using the stereo confidences. Therefore, when the stereo confidence is small on a pixel, the stereo estimation is smoothed more using neighbourhood estimations. In our experiments, we showed that our strategy outperforms SO approach with constant energy penalties significantly.