Bug 56963 - FILEOPEN, EDITING: Stability degrades with large document
Summary: FILEOPEN, EDITING: Stability degrades with large document
Status: NEW
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: Writer (show other bugs)
Version: 4.3.2.2 release
Hardware: x86-64 (AMD64) Linux (All)
: high major
Assignee: Not Assigned
QA Contact:
URL: http://bit.ly/1pLAcSK
Whiteboard:
Keywords: have-backtrace
Depends on:
Blocks:
 
Reported: 2012-11-10 19:14 UTC by wa4otj
Modified: 2014-10-15 21:38 UTC (History)
3 users (show)

See Also:
i915 platform:
i915 features:


Attachments
console bt (5.35 KB, text/plain)
2014-10-15 18:13 UTC, Julien Nabet
Details
Valgrind trace (6.19 KB, application/x-bzip)
2014-10-15 21:21 UTC, Julien Nabet
Details

Description wa4otj 2012-11-10 19:14:57 UTC
I have a rather large document (270+ pages) that has been years in development.  It has tons of footnotes, text boxes, and inserted images, both local images and links.  The ODT file is about 94 MB.  I have experienced problems for some time, but they are intermittent, come and go, and have mostly been merely an annoyance.  However, as the document continues to grow they have become more and more severe.  The problem appears in several recent version of LibreOffice, as well as OpenOffice and appears on both Mac and PC.

I spent some time with xmllint dissecting the document for errors that might contribute to the issue, finding no problems, no errors.  I have also experimented with splitting the document into a master document with various sub-documents, but similar problems still dog my work, and the master/sub document configuration gives me formatting issues that I have not been able to overcome as yet, but that's another discussion.

The primary symptom is that writer suddenly becomes unresponsive.  This seems to coincide with any operation that might cause a repagination of the entire document.  Usually, hitting Tools|Update|Update All will provoke the issue about 90% of the time, but then other times the update occurs without problems.  Attempting to export as PDF also often does so.  Lately it has been crashing on a file|save operation, which is REALLY annoying.  I have left it sitting at times for hours, hoping it might recover, but it does not, the only recovery being to force a close and then reopen and recover the document.  Usually this means recent changes are lost, which is very annoying.

Occasionally, rather than simply lock up and become totally unresponsive, writer just quits unexpectedly.  Again, recent changes are usually lost.

A few times I have simply opened the document, and then had to leave my desk for a time.  Without my ever touching a keystroke after opening the document, I find it crashed when I return.  I believe it will eventually crash if simply left alone.

I need help in getting to the bottom of the trouble.  There are two aspects.  First determine precisely what it is about this document that provokes the misbehavior.  Perhaps there are XML errors that I have missed when looking at it with xmllint.  Any advice on what to look for, how to use xmllint to best troubleshoot, or recommendation for a better tool is most welcome.  I am however inclined to believe it is simply the sheer size of the document, along with the number of insertions.

Secondly, even if the problem is due to document errors, clearly LibreOffice is not dealing well with the errors.  So if we can better quantify what is going wrong, perhaps specific bug reports can address where Writer is going "off the rails" when faced with a troublesome document. 

I would like to volunteer my system (via remote access) or my document, or both to anyone from the developer community who is interested in taking a closer look.  I could obfuscate the document and post it somewhere, but as I cannot really edit it effectively any more that would be difficult.  On the other hand, there is nothing particularly secret about it, so I would trust giving it to a single developer who would promise not to disseminate it.  I can place it on Dropbox, or Google Drive and send a link via email to someone interested in troubleshooting.  I wouldn't want it widely shared, as I would hope to finish it before letting it out into the wild, but there is nothing that is really secret.

I searched the online bug reports, and do not see anything that really seems to be a duplicate.  However #56528 does seem somewhat similar, thus might have a common root-cause of the failure.

If a developer would like to take a look and try and get to the bottom of the issue, feel free to email wa4otj@yahoo.com.

Regards,
Nathan 


I am presently using Version 3.6.3.2 (Build ID: 58f22d5) on an iMac under Snow Leopard (OS X 10.6.8 ) with 4 GB memory.  Behavior is not limited to this configuration or version.
Comment 1 Roman Eisele 2012-11-11 10:43:36 UTC
Thank you very much for your bug report!

In order to reproduce the problem, I would like to test your file. I am not a developer, but an experienced QA volunteer (“bug wrangler”, as we call it); the normal procedure here is that first some QA volunteer tries to reproduce the bug, checks for dependencies, duplicates etc., and then, if successful, forwards the bug report to the responsible developer(s). I am using the same Mac OS X version as you, and a similar machine, so I guess the chances are good that I can catch the problem.

So, if you could please send the file via private e-mail to me: <mailto:bugs@eikota.de>, I will keep it confidential and try to reproduce the issue.

Thank you very much!
Comment 2 Roman Eisele 2012-11-13 13:02:08 UTC
OK, I have received Nathan’s sample file -- thank you! -- and will try to reproduce the bug. (Of course, I have already managed to reproduce it in a special case, but I want to check some other possibilities, to get a more complete image of the problems.) Please be patient, there are many bugs to deal with and I have not got much free time ...
Comment 3 Roman Eisele 2012-11-16 14:29:17 UTC
(
Sorry for the long delay! However, reproducing issues which make it necessary
to let an application run for hours (repeatedly!) in the foreground to see
if it will finally crash, or finally recover from the apparent freeze, is not what I can do by the way in gaps of my daytime occupations ;-)
)

Again: thank you very much for your bug report!

At least one problem with the file in question seems nearly 100% REPRODUCIBLE
for me (tested with LibreOffice 3.6.3.2 on Mac OS X 10.6.8, Intel):

> The primary symptom is that writer suddenly becomes unresponsive.
> This seems to coincide with any operation that might cause a repagination
> of the entire document.  Usually, hitting Tools|Update|Update All will
> provoke the issue about 90% of the time [...]

Steps to reproduce:
0) Reset your LibreOffice user profile (to preclude any influence
   of local settings)
1) Start LibreOffice.
2) Open the file.
3) Without waiting too long, select “Tools > Update > Update All”
   from the menu bar.
4) Let LibreOffice work; it seems frozen, but I wanted to test
   if it would finally recover from this state.
   I haven’t done exact time measurements,
   but on my MacBook Pro 2010 with 2.53 GHz Intel Core i5, 4 GB RAM
   and SSD hard drive this will take about 45 min.
-> Finally, LibreOffice crashes.

I will attach the log file created by Mac OS X for the crash. It is of impressive simplicity: Thread 0 crashed with EXC_BAD_ACCESS (SIGSEGV),
and the stack trace for Thread 0 just reads:

0   libswlo.dylib  0x211b7e60 SwTxtFrm::HideAndShowObjects() + 608
1   libswlo.dylib  0x211b7e78 SwTxtFrm::HideAndShowObjects() + 632
................ the last line repeated 510 times ................

This crash is already reproducible by the steps given above with LibreOffice 3.3.0, and also with every version I have tested (3.4.0, 3.5.0, 3.6.0.4).
So this problem was probably inherited from OpenOffice.org.

It is still present in the current daily build:
LOdev 4.0.0.0.alpha0+ (Build ID: ed8067; pull time: 2012-11-15 03:54:19).


I hope that further research will help to isolate what exactly causes so much work for LibreOffice that it finally crashes ;-) But for now: at least this apparent freeze and final crash are really almost 100% reproducible.
Comment 4 David 2012-12-28 17:02:25 UTC
I have a complex document that for some reason has started to do exactly this also.  Performing an update on the page formatting will cause it to crash every time.  If a manual update isn't performed then at some point it seems like it will do it on its own and crash after a period of time trying to update.
Comment 5 David 2012-12-28 17:58:41 UTC
If the displaying of comments is turned off in options it seems to function normally (at least so far).  Also a slight delay in the display of characters when typing seems to be gone also.  It would seem that when the comments are being displayed extra work is being done which shouldn't be happening.
Comment 6 wa4otj 2012-12-28 18:05:11 UTC
Perhaps I'm missing something, but I have no comments in my Document, and under the view menu, "Comments" is grayed out.  Is there somewhere else you are saying to turn off comments?
Comment 7 wa4otj 2012-12-28 18:09:32 UTC
Oh, I see, under Preferences|View there is a comments checkbox....  I guess that is what you mean.  That is unchecked also...
Comment 8 David 2012-12-28 20:27:12 UTC
When I just unchecked Comments under the View menu I still experienced problems.  But I haven't experienced any problems yet when I unchecked the comments box in Tools - Options - LibreOffice Writer - View.
Comment 9 wa4otj 2012-12-28 23:30:51 UTC
A bit of clarification:  The Mac and PC versions have the option in very different places.  Took me a bit of poking around to find it.  But once I found it, and turned it off, and double checked, the answer is nope, it still fails even with the comments disabled.  It does seem to have a bit of an impact, whihc I presume is due to a shift in workload.  But it still dies.
Comment 10 David 2012-12-29 16:12:20 UTC
From your first message I do notice there are some differences between our documents. My document runs about 3000 pages but is only about 2.5MB in size. Yours is 94MB with about 270 pages. Yours also has text boxes and images which mine doesn't have. 

I would try turning off some of the other display options and see if that makes a change when a page formatting update is done. My document isn't using any of those other items so that may be why mine works when I turn off the commenting. It may be you can turn them off, do an update, then turn them back on again.  Of course in my experience with the comments, if that works it will need to be done again the next time the document is opened.  But it may give an indication where the problem is and a temporary work around until it's fixed.
Comment 11 wa4otj 2013-01-01 17:56:47 UTC
I have tried many variations, turning off features, changing this and that, without impacting the issue.  I am still able to edit the document, and add new material, but I am no longer able to produce a PDF or update the indices.  At first, it would only crash occasionally on updates, then it became common, but persistence would allow me to do a good update.  Now I am increasingly stuck.  Writer is becoming unusable for me.

My partners in the project are pressuring me to produce a publishable eBook soon, or at least an updated PDF.  I am increasingly feeling in a quandary as to what to do.  Perhaps I shall have to try to move to Word, or other system.  I shudder at the thought of the re-working that might require.

Is there anything else I can do to help get to the bottom of the problem?  I really need to get beyond this issue...

Thanks,
Nathan
Comment 12 David 2013-01-02 09:16:44 UTC
Have you tried version 3.4.6 with the display options off?  I haven't actually tried seeing what happens in later versions with my document and this error because of another bug which is preventing me from upgrading. Another thing I've found that worked until I closed the document and re-opened it was to create a new document and insert the old one using the File command from the insert menu.
Comment 13 wa4otj 2013-01-02 14:45:04 UTC
I am using the latest.  Display options do not really affect it much.  I will try the "Insert" method.


 
---------- 8< ----------


I use my ham call, WA4OTJ as my email ID on many different email providers.  In the event that Yahoo Email is unavailable I can be reached as wa4otj@gmail.com or wa4otj@hotmail.com


________________________________
 From: "bugzilla-daemon@freedesktop.org" <bugzilla-daemon@freedesktop.org>
To: wa4otj@yahoo.com 
Sent: Wednesday, January 2, 2013 1:16 AM
Subject: [Bug 56963] FILEOPEN, EDITING: Stability degrades with large document
 

 
Comment # 12 on bug 56963 from David  
Have you tried version 3.4.6 with the display options off?  I haven't actually
tried seeing what happens in later versions with my document and this error
because of another bug which is preventing me from upgrading. Another thing
I've found that worked until I closed the document and re-opened it was to
create a new document and insert the old one using the File command from the
insert menu.
________________________________
 You are receiving this mail because: 
	* You reported the bug.
Comment 14 David 2013-01-02 15:01:52 UTC
I really suggest going back to a pre-3.5.0 version and trying it without the display options.
Comment 15 Julien Nabet 2013-01-30 06:04:37 UTC
wa4otj: Could you attach the file or put it available somewhere so more people may test it?
Roman gave interesting info (I think about SwTxtFrm::HideAndShowObjects) but it could be interesting to retrieve backtraces at random. Moreover, devs would be able to know if a patch would solve or not the problem by trying to reproduce.
Comment 16 wa4otj 2013-01-30 06:21:41 UTC
Thanks for your interest.

I placed an obfuscated version of the large file on my Dropbox.  You can retrieve it here:

https://dl.dropbox.com/u/11336636/Gregory%20Family%20Origins%20-%20Submission.odt

I also have another file, a chapter excerpted from the larger file.  There are two versions of it, one that triggers the problem, or at least a somewhat similar mis-behavior, and one that does not, yet nothing appears much different between them.  

They are at:

https://dl.dropbox.com/u/11336636/Gregory%20Family%20Origins12y.odt

https://dl.dropbox.com/u/11336636/Gregory%20Family%20Origins12z.odt

I am not sure they are exactly the same problem, but may be related.  In any case, the fact that one file provokes problems, and the other does not is very interesting...

Good luck
Comment 17 Björn Michaelsen 2014-01-17 00:43:40 UTC
(This is an automated message.)

LibreOffice development currently prioritizes bugs with the so called MAB (most annoying bugs) -- as this bug has not run through that process (including writing a short rationale for this bug being a candidate and other who are watching the tracker bug silently approving that rationale etc.) its priority is set to high. Note this is effectively no change in the urgency assigned to this bug, as we are currently not making a difference between high and highest and severity is untouched.

You can find out more about MABs and how the process works by contacting libreoffice qa on irc:

 http://webchat.freenode.net/?channels=libreoffice-qa

The QA wiki page also gives you hints on how to get in contact with the team (if IRC fails you, your next best choice is the mailing list):

 https://wiki.documentfoundation.org/QA
Comment 18 CiberSheep 2014-10-07 13:46:23 UTC
I have experimenting problems with a file I've created with OpenOffice.
100 pages, images, incrusted calc, frames with tables in it... So, quite complicated file.
It opens with no problem on OpenOffice 3 on Windows. When I try to open the same file in LibreOffice 4.2.2 it frozes.

Log
http://bit.ly/1pLAcSK
Comment 19 Julien Nabet 2014-10-13 18:23:36 UTC
Guys, 1 problem = 1 bug. Indeed, it's very difficult to deal with a bugtracker and except you all have got a big file and have some stability problem, we can't guess if it's the same problem.

wa4otj: do you still have the problem with last stable LO version (4.3.2) + brand new LO profile (you can rename yours, see https://wiki.documentfoundation.org/UserProfile)? If it's the case, please make it available somewhere (sorry for the time you had done it, see comment 16, I must have missed it).

For others: give a try to what I indicated to wa4otj and if you still got have a problem, please submit a new bugtracker by following this link:
https://www.libreoffice.org/get-help/bug/
Comment 20 wa4otj 2014-10-14 00:07:00 UTC
(In reply to Julien Nabet from comment #19)
> Guys, 1 problem = 1 bug. Indeed, it's very difficult to deal with a
> bugtracker and except you all have got a big file and have some stability
> problem, we can't guess if it's the same problem.
> 
> wa4otj: do you still have the problem with last stable LO version (4.3.2) +
> brand new LO profile (you can rename yours, see
> https://wiki.documentfoundation.org/UserProfile)? If it's the case, please
> make it available somewhere (sorry for the time you had done it, see comment
> 16, I must have missed it).
> 
> For others: give a try to what I indicated to wa4otj and if you still got
> have a problem, please submit a new bugtracker by following this link:
> https://www.libreoffice.org/get-help/bug/

I am sorry to say, the last version 4.3.1.2 pretty well completely broke my ability to use LO to edit the file, as well as my faith and trust in the software.  The formatting went completely nuts, LO crashed and lost about 50 pages of prose, and about 30 pictures, and would go bonkers every time I tried to touch the file.  Thanks goodness I had plenty of good backups.  The 164 MB file dropped to 72 MB after being opened with LO.  I abandoned LO and moved completely over to cousin OO where I have been getting good results.  I have not tried 4.3.2.  LO 4.3.1.2 did seem to work well for some smaller documents I tried it on, but it has troubles with the big one.

In addition to the issues with stability and the formatting problems, the loss of the navigation buttons on the lower right side was the final straw.  With a document of this size, I NEED those buttons to effectively move around the file.  Moving them to the search bar was incredibly stupid.  The document is nearly 400 pages now.  OO is quite stable except for occasionally doing strange things when creating the PDF file, usually inserting undesired white space.

At this moment I have no plans to return to LO for the time being, at least as long as the navigation buttons remain useless and OO continues to work for me.  That might change at some point, but my interest is in working on my research project, not debugging software.  If you would like to experiment on your own with the ODT file, contact me privately and I will share.  I do not have time right now to reinstall LO and try to reproduce the problems.

Pretty much all of the problems center around images inserted into the pages.  Any image, or combination of images that covers more than about 50% of a page are trouble.
Comment 21 Julien Nabet 2014-10-15 18:09:21 UTC
Following some emails between wa4otj and me, I put this one to NEW for the moment.
Comment 22 Julien Nabet 2014-10-15 18:13:58 UTC
Created attachment 107898 [details]
console bt

On pc Debian x86-64 with master sources updated today, I had a crash by doing this:
make debugrun
run --writer
open file
scroll down about 20/30 pages until crash
Comment 23 Julien Nabet 2014-10-15 21:21:37 UTC
Created attachment 107900 [details]
Valgrind trace
Comment 24 Julien Nabet 2014-10-15 21:38:25 UTC
wa4otj: perhaps just a message in a bottle: http://nabble.documentfoundation.org/Dev-f1639786.html


Use of freedesktop.org services, including Bugzilla, is subject to our Code of Conduct. How we collect and use information is described in our Privacy Policy.