Bug 37859 - Odb data copied to Calc showed wrong encoding in Windows
Summary: Odb data copied to Calc showed wrong encoding in Windows
Status: NEW
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: Database (show other bugs)
Version: 3.4.4 release
Hardware: x86 (IA32) Windows (All)
: medium normal
Assignee: Not Assigned
QA Contact:
URL:
Whiteboard:
Keywords:
: 79631 (view as bug list)
Depends on:
Blocks:
 
Reported: 2011-06-02 08:53 UTC by Cheng-Chia Tseng
Modified: 2015-01-03 17:41 UTC (History)
6 users (show)

See Also:
i915 platform:
i915 features:


Attachments
Windows platform. (89.68 KB, image/png)
2011-06-02 08:53 UTC, Cheng-Chia Tseng
Details
Linux platform (353.03 KB, image/png)
2011-06-02 08:53 UTC, Cheng-Chia Tseng
Details

Description Cheng-Chia Tseng 2011-06-02 08:53:07 UTC
Created attachment 47456 [details]
Windows platform.

There is a user complained that copying Odb file (using Traditinal Chinese) data to Calc has characters which are encoed wrongly.

You can get the file at here: https://docs.google.com/leaf?id=0B9JKiYcC-SFQMDIwNjA5ODItMzFkOC00M2M3LTgxYTUtMDc4NmEwYzY2YWMy&hl=en_US&authkey=CMqpnt0H

I tested LibO 3.4 RC2, the problem do exist. But the problem does not occur on LibO 3.3.2 Linux platform.

See the attachments for output of Windows platform (having bug), and Linux platform (having no bug).
Comment 1 Cheng-Chia Tseng 2011-06-02 08:53:48 UTC
Created attachment 47457 [details]
Linux platform
Comment 2 Cheng-Chia Tseng 2011-06-02 08:58:36 UTC
In all, the copying action from Base will produce characters which are not eoncoded well (http://en.wikipedia.org/wiki/Mojibake) in Calc of Windows, but not in Calc of Linux.
Comment 3 Andras Timar 2011-06-06 23:39:45 UTC
I could not reproduce the error on Windows with LibreOffice 3.4.0. Can you please give the steps you made? It works well via Data Sources (F4) in Calc, also copy & paste strings from Base works.
Comment 4 Cheng-Chia Tseng 2011-06-07 07:21:53 UTC
I use the mehtod in this blog post: http://openoffice.blogs.com/openoffice/2007/04/farrrrrr_simple.html

Go to "Table" first, then right click the Data I would like to "Copy" to Calc. This would create Data in wrong encoding.

Plus, drag the Data icon directly and drop to Calc, the problem can also be avoided.
Comment 5 Rainer Bielefeld Retired 2011-06-10 02:57:55 UTC
RC2 is bit by bit identical with release version, so separate items in the version picker are useless. Changes have been discussed with Michael Meeks.
Comment 6 Cheng-Chia Tseng 2011-07-29 04:10:22 UTC
It is not fixed yet in 3.4.2 RC3.

Andras, could you verify this bug again? Thanks.
Comment 7 Björn Michaelsen 2011-09-23 11:41:18 UTC
*** Bug 40766 has been marked as a duplicate of this bug. ***
Comment 8 Julien Nabet 2011-11-26 05:29:13 UTC
Is the bug still there on 3.4.4 ?
What's the font and encoding to use to test ?
Comment 9 Cheng-Chia Tseng 2011-11-28 02:34:23 UTC
It is still there.

I don't know actually the encoding is, but Chinese (Traditional) Windows usually use "big5" as default.

The problem only exist when you use the "Copy" item of the context menu to paste on Calc.

However, it you open Calc first, and drag the table in odb file directly to the Calc, and everything is fine. This is the expected output.
Comment 10 sasha.libreoffice 2011-12-21 05:56:31 UTC
Repruduced.
Windows XP 32 bit LibO 3.4 russian language
Problem is more interesting than expected. When I copied table from database and inserted it into Calc, it inserted without problem. Then I changed one field in database to text on russian (Cyrillic) , copied table and inserted to Calc. All except russian inserted ok, but russian text looks wrong.
To reproduce this looking of russian text manually, I have saved document with russian text as text in Windows encoding, then opened it in webbrowser and changed encoding to ISO-8859-1. Characters become looking as described above problem.
Therefore it is locale-specific problem.
This bug resembles Bug 39890
Comment 11 Björn Michaelsen 2011-12-23 12:06:41 UTC
[This is an automated message.]
This bug was filed before the changes to Bugzilla on 2011-10-16. Thus it
started right out as NEW without ever being explicitly confirmed. The bug is
changed to state NEEDINFO for this reason. To move this bug from NEEDINFO back
to NEW please check if the bug still persists with the 3.5.0 beta1 or beta2 prereleases.
Details on how to test the 3.5.0 beta1 can be found at:
http://wiki.documentfoundation.org/QA/BugHunting_Session_3.5.0.-1

more detail on this bulk operation: http://nabble.documentfoundation.org/RFC-Operation-Spamzilla-tp3607474p3607474.html
Comment 12 Björn Michaelsen 2011-12-23 17:01:30 UTC
needinfo keyword redundant by needinfo status.
Comment 13 Cheng-Chia Tseng 2012-01-18 05:31:59 UTC
The bug is still there.

Using the mehtod in this blog post:
http://openoffice.blogs.com/openoffice/2007/04/farrrrrr_simple.html

Go to "Table" first, then right click the Data I would like to "Copy" to Calc.
This would create Data in wrong encoding.

Plus, drag the Data icon directly and drop to Calc, the problem can also be
avoided.
Comment 14 Urmas 2012-11-23 08:57:30 UTC
The data formats for both RTF and HTML contain certain text in a system default encoding, but it is either marked as a charset 0 or windows-1252.
Comment 15 sasha.libreoffice 2012-11-23 12:04:19 UTC
Possibly related to Bug 36144
Comment 16 Urmas 2012-11-23 19:57:32 UTC
The culprit probably is

/core/dbaccess/source/ui/misc/TokenWriter.cxx:421

Also, the logic here seems strange to me:

/core/svtools/source/svrtf/rtfout.cxx:118
Comment 17 Julien Nabet 2014-07-23 19:56:03 UTC
Any update with last LO stable version, 4.2.5?
Comment 18 Cheng-Chia Tseng 2014-08-30 08:30:02 UTC
Bug existed, reproducible on 4.3.0.
Comment 19 Julien Nabet 2014-08-30 08:36:51 UTC
Cheng-Chia: Thank you for your feedback, put it back to NEW.
Comment 20 Dimitris Xenakis 2014-10-08 11:18:06 UTC
I can confirm the issue, exactly as described elsewhere in this thread, except this time for Greek fonts. 
Libreoffice Version: 4.3.0.4
Build ID: 62ad5818884a2fc2e5780dd45466868d41009ec0 on Windows 7 Pro
Comment 21 Dimitris Xenakis 2014-10-08 11:50:36 UTC
Please see this Bug 79631 where it seems that Dominik has tackled the issue in version 4.4.0.0.alpha0+ . And apologies for having forgotten my own submission...
Comment 22 Julien Nabet 2014-10-09 21:28:34 UTC
On pc Debian x86-64 with 4.3.2 Debian package, I don't reproduce the very similar fdo#79631 put in See Also

Cheng-Chia: Since, I don't have Google account, could you give a new try with 4.3.2 version?
Comment 23 Cheng-Chia Tseng 2014-10-10 03:06:26 UTC
As reported, this bug only existed on Windows platform. Linux is not affected.

You can get the file at https://www.dropbox.com/s/tbb7bgffees5igj/%E5%B7%A5%E7%A8%8B%E7%AE%A1%E7%90%86%E8%B3%87%E6%96%99%E5%BA%AB2.odb?dl=0

Tested with version 4.3.2 on Windows 7 64bit, this long life bug exists still.
Comment 24 Dimitris Xenakis 2014-10-10 08:45:48 UTC
If one needs another file for testing, the attachment in bug 79631 does the trick. It is a small odb file containing a single table with fonts in various encodings exhibiting the issue. Here is the link: https://bugs.freedesktop.org/attachment.cgi?id=100392
Comment 25 Urmas 2014-10-10 16:25:49 UTC
The text in the 'system' encoding on Windows is copied to RTF as ANSI. It is marked by a font with \charset0. That is what causes the issue.
Comment 26 Urmas 2014-10-10 16:27:19 UTC
*** Bug 79631 has been marked as a duplicate of this bug. ***
Comment 27 Urmas 2014-10-10 16:53:26 UTC
If you want to reproduce it, you will need a Windows OS set to any legacy codepage than 1252.

P.S. Bugs do not fix themselves by magic after two years.
Comment 28 Julien Nabet 2014-10-13 18:13:33 UTC
Urmas: bug may indeed be "magically" fixed sometimes when:
- a similar bug has been fixed
- some code part has been redesigned
- a problem indicated by code analyzers (like coverity scan, cppcheck and other), which was the root cause of the bug, has been fixed
etc.
Of course, I wouldn't be able to give you any probalities but it does happen sometimes! :-)
Comment 29 Alex Thurgood 2015-01-03 17:41:13 UTC
Adding self to CC if not already on


Use of freedesktop.org services, including Bugzilla, is subject to our Code of Conduct. How we collect and use information is described in our Privacy Policy.