Bug 503

Summary: libX11: PtsToRegion corrupts memory if called with numFullPtBlocks = 0 and iCurPtBlock = 0
Product: xorg Reporter: Dr. Thomas Gern <thomas.gern>
Component: Lib/XlibAssignee: Jim Gettys <jg>
Status: CLOSED FIXED QA Contact:
Severity: normal    
Priority: high CC: luik
Version: unspecifiedKeywords: patch
Hardware: x86 (IA32)   
OS: Linux (All)   
Whiteboard: 2011BRB_Reviewed
i915 platform: i915 features:
Attachments:
Description Flags
patch to fix the problem locally none

Description Dr. Thomas Gern 2004-04-19 19:06:49 UTC
The function PtsToRegion is called internally from XPolygonRegion.
If the Polygon consists e.g of the following points:
####### point #0 at 990, 825
####### point #1 at 991, 825
####### point #2 at 990, 825

the function PtsToRegion is called with numFullPtBlocks = 0 and iCurPtBlock = 0. 

This leads to numRects=0 and causes a memory corruption, because the
following call in PolyReg.c, line 410ff
    if (!(reg->rects = (BOX *)Xrealloc((char *)reg->rects, 
	    (unsigned) (sizeof(BOX) * numRects))))  {
	Xfree(prevRects);
	return(0);
    }

performs the deallocation in Xrealloc and again in Xfree
-> the data structures of the allocator are corrupted!

The reason for this behaviour is desribed in the man page of
realloc:
"if size is equal to zero, the call is equivalent to free(ptr)."

From my point of view, this behaviour is a problem of all
X-functions using Xrealloc because it is assumed that 
it never returns NULL on success (see definition of macro Xrealloc).

There are two solutions:
- global solution: either change macro realloc 
  (I don't know if there are any draw backs of this solution, so 
   I did not try that one...)
- local solution:
  only perform deallocation if numRects != 0
  this gives:
    if (!(reg->rects = (BOX *)Xrealloc((char *)reg->rects, 
	    (unsigned) (sizeof(BOX) * numRects))))  {
	if (numRects != 0) {
	  Xfree(prevRects); 
	}
	reg->size = 0;
	return(0);
    }

Note: it is obvious that the coordinates of the example are not typical,
nevertheless the function should never corrupt the memory....
Comment 1 Dr. Thomas Gern 2004-04-19 19:17:36 UTC
Created attachment 212 [details] [review]
patch to fix the problem locally

This is the fix described as local solution.
Comment 2 Dr. Thomas Gern 2004-04-19 22:12:48 UTC
aeh... sorry, but not it seems to me that this
error was a self-generated one.... I included the
file PolyReg.c in my own source tree to overcome the problem
with bug-report #372 and therefore the setting
of MALLOC_0_RETURNS_NULL was not set according to the
X-compilation.

nevertheless this problem exists probably, if MALLOC_0_RETURNS_NULL is
not set... (but I have no possibility to check this...)

again, sorry for the inconvenience...

Comment 3 Jeremy Huddleston Sequoia 2011-09-24 16:37:52 UTC
Old bug.  Please try a more recent libX11.  If this is still an issue, please reopen and assign to me.

Use of freedesktop.org services, including Bugzilla, is subject to our Code of Conduct. How we collect and use information is described in our Privacy Policy.