Bug 3104

Created attachment 2509 [details] [review]
Pointerless compose data structure patch

This patch redefines the compose data structures to use indices instead of
pointers. This is necessary to be able to mmap() the internal data
structures instead of parsing the compose table.  Memory requirements are a
bit higher than before (I cannot realloc() for every single binding, so I 
multiply by 1.5 each time space runs out), but performance is equivalent.

Tested this patch with all locales and input methods known to me and finally
works with SCIM again as well.	Cache files are endian-safe and
typesize-safe (both encoded into filenames) and versioned.


There are a couple of issues to be thought about, though:

- I'm not exactly sure about the differences between INT32 and BITS32 types.  
  I chose these types for creating fixed sized structures.

- I defined the type of modifiers in the structure to be BITS32. It was just
  'unsigned' before. Yes, no int or anything else.

- How much memory should I allocate in the very beginning? I chose numbers 
  so that the 'C' locale bindings just fit nicely into memory.

- I have *not* tested the new XOpenFileMode function that is needed for 
  Windows only (_XOpenFileMode) due to lack of an according development
  environment.

- Memory that has been allocated during the creation of the structure is not
  freed again. This could happen, when the Compose files contain errors or
  overwrites already defined keysyms.

- The Thai imput method is inherrently broken, and we should think about
  dropping it altogether.

Created attachment 2510 [details] [review]
Compose cache patch

This is the compose cache itself.

The parsed compose tables are stored in /var/X11R6/compose-cache/ (global
cache) or ~/.compose-cache/ (local cache), respectively. Files in the local
cache expire once per day. File dates are compared with the compose table
source files as well. suid and sgid programs are not allowed to use the
local cache.

Caches currently depend on the used locale (more exactly: on the encoding),
but this isn't respected in the cache file format. This has to be changed.
So at least for global cache files, only UTF8 compose files should be
considered.

For user caches this is less problematic as clashes might only occure if the
user
a) has created a .compose-cache
b) uses different locales at the same time
c) these locales use the same compose table 
d) parsing of the compose table differs for the used
   locales

Issues to be discussed:

- The per-user cache directory ~/.compose-cache/ is not created if it does
  not exist. In this case non-system compose tables are not cached at all.
  Should it be created automatically if it does not exist?
  I personally doubt this is really usefull.

Created attachment 2511 [details] [review]
Helper program for creating global cache files

This tool creates a cache file for a given locale and compose file.
Unfortunately, due to the way XIM is integrated into libX11, a Xserver
has to be running in order to create a cache. If somebody sees a clearer
way to deal with this you're very welcome to change it.

For distributors I also add an example script below that shows how global cache

files for the three UTF8 compose tables can created using Xvfb. This is only a
skeleton, though.

Currently paths are hardcoded. Also note that compose tables for non-UTF8
locales currently shouldn't be cached globaly, as the encoding may differ
with different locales. This is something to be addressed.


------> CUT <-------
#!/bin/sh

if [ $# -gt 1 ]; then
  echo "$0 [x11-root]"
  exit 0
fi

if [ $# -eq 1 ]; then
  ROOT=$1
fi

mkdir -p $ROOT/var/X11R6/compose-cache

tmpfile=$(mktemp /tmp/Xvfb.log.XXXXXXXXXX)
$ROOT/usr/X11R6/bin/Xvfb \
  -fp $ROOT/usr/X11R6/lib/X11/fonts/misc/ \
  -sp $ROOT/etc/X11/xserver/SecurityPolicy \
  -co $ROOT/usr/X11R6/lib/X11/rgb \
  :99 &> $tmpfile &
trap "kill $!; rm $tmpfile || true" EXIT

export DISPLAY=:99
export LD_LIBRARY_PATH=$ROOT/usr/X11R6/lib64:$ROOT/usr/X11R6/lib
export XLOCALEDIR=$ROOT/usr/X11R6/lib/X11/locale/
sleep 5
$ROOT/usr/X11R6/bin/xbiff &> /dev/null &

pushd $ROOT/

usr/X11R6/bin/mkcomposecache "en_US.UTF-8"
"usr/X11R6/lib/X11/locale/en_US.UTF-8/Compose" "var/X11R6/compose-cache"
"/usr/X11R6/lib/X11/locale/en_US.UTF-8/Compose"
usr/X11R6/bin/mkcomposecache "pt_BR.UTF-8"
"usr/X11R6/lib/X11/locale/pt_BR.UTF-8/Compose" "var/X11R6/compose-cache"
"/usr/X11R6/lib/X11/locale/pt_BR.UTF-8/Compose"
usr/X11R6/bin/mkcomposecache "el_GR.UTF-8"
"usr/X11R6/lib/X11/locale/el_GR.UTF-8/Compose" "var/X11R6/compose-cache"
"/usr/X11R6/lib/X11/locale/el_GR.UTF-8/Compose"

popd

chmod 444 $ROOT/var/X11R6/compose-cache/*

Egbert asked for it :)

Adding patch keyword. Any plan for merging?

We're in the middle of a release phase. After that I plan to address a couple of
minor issues I'm seeing with my patch, and merge it.

7.1 is out now, so, ping?

Next thing to work on for me. Might be distracted a bit right now due to SLES 10.
The code still needs some cleanup, especially cache file names should include
encoding.

Commited & released.

Out of interest, why was the actual compose cache patch applied, but not the patch which adds the helper program (mkcomposecache)?