Bug 97967

There is a problem creating the folders I'm not sure why. Do you have some kind of odd user setup?

Either way I don't think this should be a blocked for 13.0 because nobody should be building with this enabled as it still doesn't do anything.

maybe it does not implement '-p' for the mkdir? My original cache set did this, if wanted some code can be taken from here:

https://cgit.freedesktop.org/~tpalli/mesa/log/?h=automatic_cache_mgmt

following patches:

mesa: add _mesa_mkdir helper to imports.h
mesa/program: add disk cache functionality

makes parent directories if needed and is more windows friendly (AFAIK apart from mkdir and mmap it looks like cache code should work on windows).

The "Failed to create foo" messages are expected and deliberate.

The actual issue is the next lines:

Error: Test 'cache_put eviction with MAX_SIZE=1K' failed: Expected=1, Actual=2
Error: Test 'eviction after overflow with MAX_SIZE=1M' failed: Expected=2, Actual=3

Currently we have 7 (seven) different codepaths for the SHA implementation (as provided by 8 libraries). My local work to resolve that ran into similar issues. That said the following come to mind:
 - Buggy sha implementation - play with --with-sha1 + libc|libmd|libnettle|libgcrypt|libcrypto|libsha1|CommonCrypto|CryptoAPI
 - Concurrency issues - does 5c73ecaac487eba36e15f22be2e9396c4a0ffe46 help ?
 - Mesa bugs ?

Fwiw I've ran into identical issue whist importing a SHA1 implementation.

The issue here is buggy SHA1 implementation. As such if you print the result of _mesa_sha1_compute and match that across sha1 implementation you'll see it first hand.

Thus for this bug the cache_test.c comment "For this test, we force this signature to land in the same directory as the original blob first written to the cache," is not true, leading to the interesting experience.

Fwiw I've imported OpenBSD's implementation [1]. It works great, saves us a bit of code, configure script and toggles.

[1] https://patchwork.freedesktop.org/patch/133113/

Should no longer be an issue, since

commit d1efa09d342bff3e5def2978a0bef748d74f9c82
Author: Emil Velikov <emil.velikov@collabora.com>
Date:   Fri Jan 13 16:51:31 2017 +0000

    util: import sha1 implementation from OpenBSD

mesa: 3eada948a05d34fabfcfac69d6c33708cb1cf740 (master 17.1.0-devel)


====================================================
   Mesa 17.1.0-devel: src/compiler/test-suite.log
====================================================

# TOTAL: 10
# PASS:  9
# SKIP:  0
# XFAIL: 0
# FAIL:  1
# XPASS: 0
# ERROR: 0

.. contents:: :depth: 2

FAIL: glsl/tests/cache-test
===========================

Failed to create ./cache-test-tmp/xdg-cache-home for shader cache (No such file or directory)---disabling.
Failed to create ./cache-test-tmp/mesa-glsl-cache-dir for shader cache (No such file or directory)---disabling.
Error: Test 'disk_cache_put eviction with MAX_SIZE=1K' failed: Expected=1, Actual=2
Error: Test 'eviction after overflow with MAX_SIZE=1M' failed: Expected=2, Actual=3

The real issue is these MAX_SIZE limits are far too small. Arguably we should handle it anyway or maybe apply a minimum size.

(In reply to Timothy Arceri from comment #7)
> The real issue is these MAX_SIZE limits are far too small. Arguably we
> should handle it anyway or maybe apply a minimum size.

hmm let me share my theory .. MAX_SIZE limit set by this test is small intentionally to be able to force eviction to happen. There is some intent in the test to evict exact wanted entry BUT for some reason picking that exact entry on CentOS fails. My theory is that in CentOS environment compiler generates such code that our integrated sha1 implementation will produce different results than on system where test was generated.

To check the theory in comment 8 just add a couple of printfs in test_put_and_get() and/or get_cache_file().

Vinson can you give this a try - it will save us all a fair bit of guesswork/time.

Can someone provide a patch with the print statements wanted to test?

FRom the gentoo bug:

"This only happens with ABI_X86=32, the test passes for native abi."

Created attachment 129357 [details] [review]
Some debug printfs

Adding a reference to the Gentoo bug and the patch in question.
Once executed the output should be as below. I've intentionally left the whole sha1 as a complete reference point.


Failed to create ./cache-test-tmp/xdg-cache-home for shader cache (No such file or directory)---disabling.
Failed to create ./cache-test-tmp/mesa-glsl-cache-dir for shader cache (No such file or directory)---disabling.
_mesa_sha1_compute ca db d9 c9 9c d7 e1 b4 e0 a5 6f cf 25 8e 83 b5 9f 74 5a 56
disk_cache_put: get_cache_file() returned "./cache-test-tmp/mesa-glsl-cache-dir/ca/dbd9c99cd7e1b4e0a56fcf258e83b59f745a56".
NOTE the directory name wrt sha1[0].

_mesa_sha1_compute 4d 78 cc 97 5b 41 97 6b b e3 58 2c 90 bf bd 6 26 a f4 d8
disk_cache_put: get_cache_file() returned "./cache-test-tmp/mesa-glsl-cache-dir/4d/78cc975b41976b0be3582c90bfbd06260af4d8".
NOTE the directory name wrt sha1[0].

_mesa_sha1_compute 60 ca cb f3 d7 2e 1e 78 34 20 3d a6 8 3 7b 1b f8 3b 40 e8
test_put_and_get: Adjusting the first byte of one_KB_key[] the to 0xca.
This is to ensure that the correct folder is picked "ca/" for the eviction that follows.
disk_cache_put: get_cache_file() returned "./cache-test-tmp/mesa-glsl-cache-dir/ca/cacbf3d72e1e7834203da608037b1bf83b40e8".
NOTE the directory name wrt sha1[0].

disk_cache_put: get_cache_file() returned "./cache-test-tmp/mesa-glsl-cache-dir/ca/dbd9c99cd7e1b4e0a56fcf258e83b59f745a56".
NOTE the directory name wrt sha1[0].

disk_cache_put: get_cache_file() returned "./cache-test-tmp/mesa-glsl-cache-dir/4d/78cc975b41976b0be3582c90bfbd06260af4d8".
NOTE the directory name wrt sha1[0].

_mesa_sha1_compute 3b 71 f4 3f f3 f 4b 15 b5 cd 85 dd 9e 95 eb c7 e8 4e b5 a3
disk_cache_put: get_cache_file() returned "./cache-test-tmp/mesa-glsl-cache-dir/ca/71f43ff30f4b15b5cd85dd9e95ebc7e84eb5a3".
NOTE the directory name wrt sha1[0].

Another attempt to add the Gentoo bug report.

Semi-random thoughts which came to mind. Feel free to pursue if interested - I'm sidetracked with something else atm. 
 - endianess issue ?
 - fs is missing/buggy/etc. flock and friends ?
 - sizeof(void *) related ?

Here is make check failure with the debug statements from attachment 129357 [details] [review].

FAIL: glsl/tests/cache-test
===========================

Failed to create ./cache-test-tmp/xdg-cache-home for shader cache (No such file or directory)---disabling.
Failed to create ./cache-test-tmp/mesa-glsl-cache-dir for shader cache (No such file or directory)---disabling.
_mesa_sha1_compute ca db d9 c9 9c d7 e1 b4 e0 a5 6f cf 25 8e 83 b5 9f 74 5a 56
disk_cache_put: get_cache_file() returned "./cache-test-tmp/mesa-glsl-cache-dir/ca/dbd9c99cd7e1b4e0a56fcf258e83b59f745a56".
NOTE the directory name wrt sha1[0].

_mesa_sha1_compute 4d 78 cc 97 5b 41 97 6b b e3 58 2c 90 bf bd 6 26 a f4 d8
disk_cache_put: get_cache_file() returned "./cache-test-tmp/mesa-glsl-cache-dir/4d/78cc975b41976b0be3582c90bfbd06260af4d8".
NOTE the directory name wrt sha1[0].

_mesa_sha1_compute 60 ca cb f3 d7 2e 1e 78 34 20 3d a6 8 3 7b 1b f8 3b 40 e8
test_put_and_get: Adjusting the first byte of one_KB_key[] the to 0xca.
This is to ensure that the correct folder is picked "ca/" for the eviction that follows.
disk_cache_put: get_cache_file() returned "./cache-test-tmp/mesa-glsl-cache-dir/ca/cacbf3d72e1e7834203da608037b1bf83b40e8".
NOTE the directory name wrt sha1[0].

Error: Test 'disk_cache_put eviction with MAX_SIZE=1K' failed: Expected=1, Actual=2
disk_cache_put: get_cache_file() returned "./cache-test-tmp/mesa-glsl-cache-dir/ca/dbd9c99cd7e1b4e0a56fcf258e83b59f745a56".
NOTE the directory name wrt sha1[0].

disk_cache_put: get_cache_file() returned "./cache-test-tmp/mesa-glsl-cache-dir/4d/78cc975b41976b0be3582c90bfbd06260af4d8".
NOTE the directory name wrt sha1[0].

_mesa_sha1_compute 3b 71 f4 3f f3 f 4b 15 b5 cd 85 dd 9e 95 eb c7 e8 4e b5 a3
disk_cache_put: get_cache_file() returned "./cache-test-tmp/mesa-glsl-cache-dir/ca/71f43ff30f4b15b5cd85dd9e95ebc7e84eb5a3".
NOTE the directory name wrt sha1[0].

Error: Test 'eviction after overflow with MAX_SIZE=1M' failed: Expected=2, Actual=3

Created attachment 129404 [details] [review]
Possible fix

Does this patch help?

Created attachment 129405 [details] [review]
Possible fix

Please try this one instead

Created attachment 129406 [details] [review]
Possible fix

Note to self, test patch *then* post.

Please try this one.

Applied attachment 129406 [details] [review]. make check passes.

====================================================
   Mesa 17.1.0-devel: src/compiler/test-suite.log
====================================================

# TOTAL: 10
# PASS:  10
# SKIP:  0
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 0

.. contents:: :depth: 2


Tested-by: Vinson Lee <vlee@freedesktop.org>

Should now be fixed by:

commit d7b3707c612027b354deea6bc5eae56a02d5f8d5
Author: Timothy Arceri <tarceri@itsqueeze.com>
Date:   Wed Feb 8 15:05:19 2017 +1100

    util/disk_cache: use stat() to check if entry is a directory
    
    d_type is not supported on all systems.
    
    Tested-by: Vinson Lee <vlee@freedesktop.org>
    Reviewed-by: Emil Velikov <emil.velikov@collabora.com>
    Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=97967