Created attachment 24071 [details] [review]
Patch that fixes the problem

I'm using randr 1.2 and the radeon driver with the xserver release 1.6.0.  I have one monitor rotated left, and the other rotated normal.  When I move a window in the rotated monitor, it leaves blit artifacts on the *left* side of the window.  There are no such artifacts in the un-rotated monitor.  The situation is similar when rotating one window to the right or 180 degrees.

After much hunting, I have identified the problem.  It is in the xorg-server source (both in 1.6.0 and in git), in hw/xfree86/modes/xf86Rotate.c.  The problem is in xf86RotateCrtcRedisplay (and also xf86CrtcRotate).  Here's what happens, as illustrated by example.  Suppose xf86RotateCrtcRedisplay() is told to update the framebuffer with the contents of the shadow buffer on my rotated monitor, in the logical box with bounds x1=0,y1=0,x2=100,y2=100.  This means it needs to update the *pixels* with logical coordinates (x,y) with 0 <= x < 100 and 0 <= y < 100 -- that is, the bounds of the box are inclusive in the top-left corner, exclusive on the bottom-right corner.  It then calls pixman_f_transform_bounds, which faithfully transforms the box into x1=0,y1=923,x2=100,y2=1023, then composites in the image.  The problem is, it's now composting the pixels (x,y) with coordinates 0 <= x < 100, 923 <= y < 1023, but the desired behavior is 923 < y <= 1023.  So you end up with an incorrectly updated frame buffer at y=coordinate 1023, i.e., a line on the left of your window.

I've included a patch that fixes this instance of the problem.  It just wraps pixman_f_transform_bounds by first making the bounds *inclusive* (in the example, it changes the box to x1=0,y1=0,x2=99,y2=99), does the transform, then changes it back to exclusive bounds.  I also applied this fix to xf86CrtcRotate, because crtc->bounds was being calculated incorrectly, hence my vertical monitor wasn't being updated with windows that only overlapped it with the leftmost pixel.

I know only enough about the xserver code to fix this bug, so I imagine the same error may show up in other places.  I also understand you may want to implement a better solution than what my patch does.  But it is worth some attention, since it will affect anybody with a rotated monitor and it is a quite irritating bug.