konsole

8604 Commits

3 Branches

0 Tags

11 MiB

Tag: Branch: Tree: b35919187f

Commit Graph

Author	SHA1	Message	Date
Luis Javier Merino Morán	cfff2326f9	Make non-initial Korean Hangul Jamo width 0 Korean Hangul can be represented in Unicode either as precomposed Hangul syllables, or as sequences of alphabetic components called Jamo. Syllables should occupy 2 cells (there are halfwidth variants at U+FFA0..U+FFDF). A fully decomposed syllable consists of an initial jamo (choseong - leading consonant - may be a filler U+115F), a medial jamo (jungseong - vowel - may be a filler U+1160), and an optional final jamo (jongseong - trailing consonant). Old Korean can have more than one of each of those. In any case, to make the total width 2, we assign width 2 to choseong, and 0 to jungseong and jongseong, which, absent a context-aware wcswidth, will still break with Old Korean syllables with more than one jamo for leading consonants. This aligns with glibc: commit 7a79e321c6f85b204036c33d85f6b2aa794e7c76 Author: Thorsten Glaser <tg@mirbsd.de> Date: Fri Jul 14 14:02:50 2017 +0200 Refresh generated charmap data and ChangeLog [BZ #21750] * charmaps/UTF-8: Refresh. diff --git a/localedata/ChangeLog b/localedata/ChangeLog index 04ef5ad071..9e05b4a652 100644 --- a/localedata/ChangeLog +++ b/localedata/ChangeLog @@ -1,3 +1,17 @@ +2017-07-14 Thorsten Glaser <tg@mirbsd.de> + + [BZ #21750] + * charmaps/UTF-8: Refresh. + * unicode-gen/utf8_gen.py (U+00AD): Set width to 1. + * unicode-gen/utf8_gen.py (U+1160..U+11FF): Set width to 0. + * unicode-gen/utf8_gen.py (U+3248..U+324F): Set width to 2. + * unicode-gen/utf8_gen.py (U+4DC0..U+4DFF): Likewise. + * unicode-gen/utf8_gen.py: Treat category Me and Mn as combining. + [BZ #19852] + * unicode-gen/utf8_gen.py: Process EastAsianWidth lines before + UnicodeData lines so the latter have precedence; remove hack + to group output by EastAsianWidth ranges. + [ ... snip ...] commit 6e540caa21616d5ec5511fafb22819204525138e Author: Mike FABIAN <mfabian@redhat.com> Date: Tue Jun 16 08:29:40 2020 +0200 Set width of JUNGSEONG/JONGSEONG characters from UD7B0 to UD7FB to 0 [BZ #26120] Reviewed-by: Carlos O'Donell <carlos@redhat.com> diff --git a/localedata/charmaps/UTF-8 b/localedata/charmaps/UTF-8 index 14c5d4fa33..8cce47cd97 100644 --- a/localedata/charmaps/UTF-8 +++ b/localedata/charmaps/UTF-8 @@ -48920,6 +48920,8 @@ WIDTH <UABE8> 0 <UABED> 0 <UAC00>...<UD7A3> 2 +<UD7B0>...<UD7C6> 0 +<UD7CB>...<UD7FB> 0 <UF900>...<UFA6D> 2 <UFA70>...<UFAD9> 2 <UFB1E> 0	4 years ago
Mariusz Glebocki	e74cf6c366	Use new character width code based on Unicode 11 Summary: Adds a code for getting character width togeter with LUTs generated using uni2characterwidth from Unicode 11 lists. Skin tone, flags, gender, and other emoji with and modifer are not joined (you will see e.g. a skin tone square + generic yellow emoji). I think joining them would cause problems in most editors, command line prompts, and other programs which use character width data, as the characters would behave as combining or emoji depending on context (like ligatures). Examples: * light thumb up: 👍🏻 * dark thumb up: 👍🏿 * Polish flag: 🇵🇱 This behavior is allowed: * https://unicode.org/reports/tr51/#Emoji_Modifiers_Display * https://unicode.org/reports/tr51/#Emoji_ZWJ_Sequences It is possible to add support for sequences, but those would work only for a string width functions. Some characters which can be presented as emoji are narrow (e.g. ✖️, ©️). Those characters are listed without "presentation" mode, which means they should be rendered as text by default (real presentation depends on renderer and/or font). Noto Sans Color Emoji renders them as wide, DejaVu Sans as narrow. Vim, bash and zsh treat them as narrow, so I made them narrow. https://unicode.org/reports/tr51/#Presentation_Style BUG: 396435 BUG: 378124 BUG: 392171 BUG: 339439 FIXED-IN: 18.12 Depends on D15757 Test Plan: * Look at emoji_test.txt - emojis should look "normal" (two characters width). * Look at GLASS.txt - characters width should look correct. * CharacterWidthTest should pass. * perl -XCSDL -e 'print map{chr($_), " "} 1..0xffff' Reviewers: #konsole, #vdg, hindenburg Reviewed By: #konsole, hindenburg Subscribers: hindenburg, broulik, ngraham, konsole-devel Tags: #konsole Differential Revision: https://phabricator.kde.org/D15758	8 years ago

Author

SHA1

Message

Date

Luis Javier Merino Morán

cfff2326f9

Make non-initial Korean Hangul Jamo width 0

Korean Hangul can be represented in Unicode either as precomposed Hangul
syllables, or as sequences of alphabetic components called Jamo.

Syllables should occupy 2 cells (there are halfwidth variants at
U+FFA0..U+FFDF).  A fully decomposed syllable consists of an initial
jamo (choseong - leading consonant - may be a filler U+115F), a medial
jamo (jungseong - vowel - may be a filler U+1160), and an optional final
jamo (jongseong - trailing consonant).  Old Korean can have more than
one of each of those.  In any case, to make the total width 2, we assign
width 2 to choseong, and 0 to jungseong and jongseong, which, absent a
context-aware wcswidth, will still break with Old Korean syllables with
more than one jamo for leading consonants.

This aligns with glibc:

commit 7a79e321c6f85b204036c33d85f6b2aa794e7c76
Author: Thorsten Glaser <tg@mirbsd.de>
Date:   Fri Jul 14 14:02:50 2017 +0200

    Refresh generated charmap data and ChangeLog

            [BZ #21750]
            * charmaps/UTF-8: Refresh.

diff --git a/localedata/ChangeLog b/localedata/ChangeLog
index 04ef5ad071..9e05b4a652 100644
--- a/localedata/ChangeLog
+++ b/localedata/ChangeLog
@@ -1,3 +1,17 @@
+2017-07-14  Thorsten Glaser  <tg@mirbsd.de>
+
+       [BZ #21750]
+       * charmaps/UTF-8: Refresh.
+       * unicode-gen/utf8_gen.py (U+00AD): Set width to 1.
+       * unicode-gen/utf8_gen.py (U+1160..U+11FF): Set width to 0.
+       * unicode-gen/utf8_gen.py (U+3248..U+324F): Set width to 2.
+       * unicode-gen/utf8_gen.py (U+4DC0..U+4DFF): Likewise.
+       * unicode-gen/utf8_gen.py: Treat category Me and Mn as combining.
+       [BZ #19852]
+       * unicode-gen/utf8_gen.py: Process EastAsianWidth lines before
+       UnicodeData lines so the latter have precedence; remove hack
+       to group output by EastAsianWidth ranges.
+

[ ... snip ...]

commit 6e540caa21616d5ec5511fafb22819204525138e
Author: Mike FABIAN <mfabian@redhat.com>
Date:   Tue Jun 16 08:29:40 2020 +0200

    Set width of JUNGSEONG/JONGSEONG characters from UD7B0 to UD7FB to 0 [BZ #26120]

    Reviewed-by: Carlos O'Donell <carlos@redhat.com>

diff --git a/localedata/charmaps/UTF-8 b/localedata/charmaps/UTF-8
index 14c5d4fa33..8cce47cd97 100644
--- a/localedata/charmaps/UTF-8
+++ b/localedata/charmaps/UTF-8
@@ -48920,6 +48920,8 @@ WIDTH
 <UABE8>        0
 <UABED>        0
 <UAC00>...<UD7A3>      2
+<UD7B0>...<UD7C6>      0
+<UD7CB>...<UD7FB>      0
 <UF900>...<UFA6D>      2
 <UFA70>...<UFAD9>      2
 <UFB1E>        0

4 years ago

Mariusz Glebocki

e74cf6c366

Use new character width code based on Unicode 11

Summary:
Adds a code for getting character width togeter with LUTs generated
using uni2characterwidth from Unicode 11 lists.

Skin tone, flags, gender, and other emoji with and modifer are not
joined (you will see e.g. a skin tone square + generic yellow emoji).
I think joining them would cause problems in most editors, command line
prompts, and other programs which use character width data, as the
characters would behave as combining or emoji depending on context (like
ligatures).

Examples:
* light thumb up: 👍🏻
* dark thumb up:  👍🏿
* Polish flag:    🇵🇱

This behavior is allowed:
* https://unicode.org/reports/tr51/#Emoji_Modifiers_Display
* https://unicode.org/reports/tr51/#Emoji_ZWJ_Sequences

It is possible to add support for sequences, but those would work
only for a string width functions.

Some characters which can be presented as emoji are narrow (e.g. ✖️, ©️).
Those characters are listed without "presentation" mode, which means
they should be rendered as text by default (real presentation depends on
renderer and/or font). Noto Sans Color Emoji renders them as wide,
DejaVu Sans as narrow. Vim, bash and zsh treat them as narrow, so I made
them narrow.

https://unicode.org/reports/tr51/#Presentation_Style

BUG: 396435
BUG: 378124
BUG: 392171
BUG: 339439

FIXED-IN: 18.12

Depends on D15757

Test Plan:
* Look at emoji_test.txt - emojis should look "normal" (two characters
width).
* Look at GLASS.txt - characters width should look correct.
* CharacterWidthTest should pass.
* perl -XCSDL -e 'print map{chr($_), " "} 1..0xffff'

Reviewers: #konsole, #vdg, hindenburg

Reviewed By: #konsole, hindenburg

Subscribers: hindenburg, broulik, ngraham, konsole-devel

Tags: #konsole

Differential Revision: https://phabricator.kde.org/D15758

8 years ago

2 Commits (b35919187f14f148b7838d17bd140c0afa175abb)