Encoding - Revision history

David Haslam: /* See also / Conversion to Unicode

2018-01-11T18:57:12Z

‎See also: * Conversion to Unicode

David Haslam: /* Searching in modules */ ==== Composition Exclusions ====

2017-01-19T17:06:00Z

‎Searching in modules: ==== Composition Exclusions ====

David Haslam: /* A Mac:Central Europe text decoded as UTF-8 */ example

2016-01-18T11:32:43Z

‎A Mac:Central Europe text decoded as UTF-8: example

David Haslam: /* 1. An UTF-8 text decoded as ISO-8859-1 (or ISO-8859-) /

2016-01-18T11:24:40Z

‎1. An UTF-8 text decoded as ISO-8859-1 (or ISO-8859-*)

David Haslam: /* 3. Using a font that does not include the characters you want */ rem "3. "

2016-01-18T11:20:01Z

‎3. Using a font that does not include the characters you want: rem "3. "

David Haslam: /* 2. An ISO-8859-1 text decoded as UTF-8 */ === A Mac:Central Europe text decoded as UTF-8 ===

2016-01-18T11:19:31Z

‎2. An ISO-8859-1 text decoded as UTF-8: === A Mac:Central Europe text decoded as UTF-8 ===

David Haslam: /* Common mistakes */ be displayed [http://en.wikipedia.org/wiki/Mojibake incorrectly]

2016-01-09T11:42:36Z

‎Common mistakes: be displayed [http://en.wikipedia.org/wiki/Mojibake incorrectly]

David Haslam: /* Multi-bytes encoding */ is

2015-12-09T18:50:12Z

‎Multi-bytes encoding: is

David Haslam: /* External links / [http://unicode.org/versions/Unicode8.0.0/ Unicode 8.0.0] – released 2015-06-17

2015-06-18T16:43:33Z

‎External links: * [http://unicode.org/versions/Unicode8.0.0/ Unicode 8.0.0] – released 2015-06-17

David Haslam: /* Unicode */ code-points

2014-08-05T12:40:13Z

‎Unicode: code-points

@@ Line 104: / Line 104: @@
 == See also ==
 * [[File Formats]]
 * [[Fonts]]

← Older revision		Revision as of 17:06, 19 January 2017
Line 87:		Line 87:

	The situation is made even more complicated by changes in the Unicode standard in respect of normalization. How the characters for some languages normalize to NFC has changed in later Unicode versions. One example is in the Myanmar block of the BMP. This has implications for existing and requested modules for Bible translations that use the Burmese script. i.e. for the Judson 1835 translation, and for the S'gaw Karen translation.<ref>The process of converting a string to NFC or NFD requires a stage called "canonical ordering", whereby codepoints are reordered in ascending order according to their canonical combining class [ccc].</ref><ref>Unicode normalization can easily break Biblical Hebrew text. See on page 9 in the [http://www.sbl-site.org/Fonts/SBLHebrewUserManual1.5x.pdf SBL Hebrew Font User Manual].</ref>		The situation is made even more complicated by changes in the Unicode standard in respect of normalization. How the characters for some languages normalize to NFC has changed in later Unicode versions. One example is in the Myanmar block of the BMP. This has implications for existing and requested modules for Bible translations that use the Burmese script. i.e. for the Judson 1835 translation, and for the S'gaw Karen translation.<ref>The process of converting a string to NFC or NFD requires a stage called "canonical ordering", whereby codepoints are reordered in ascending order according to their canonical combining class [ccc].</ref><ref>Unicode normalization can easily break Biblical Hebrew text. See on page 9 in the [http://www.sbl-site.org/Fonts/SBLHebrewUserManual1.5x.pdf SBL Hebrew Font User Manual].</ref>
		+
		+	==== Composition Exclusions ====
		+	There are certain precomposed '''Indic''', '''Tibetan''' & '''Hebrew''' letters that decompose but do not recompose under any normalization form. The list of these characters is given here:
		+	* [http://unicode.org/Public/UNIDATA/CompositionExclusions.txt Composition Exclusions]
		+	The decomposed forms are preferred over the precomposed forms, as indicated in:
		+	* [http://www.unicode.org/reports/tr15/#Primary_Exclusion_List_Table Primary Exclusion List Table]
		+	Composition exclusions are in these Indic scripts: Devanagari<ref>The writing system for Hindi and several other languages, including Nepali.</ref>, Bengali, Gurmukhi<ref>The writing system for Eastern Punjabi.</ref>, Oriya.<BR>
		+	These particular scripts have some canonically decomposable characters that are generally not the preferred form. This can cause serious problems for SWORD search if the front-end users are accustomed to keying these composite characters. They will simply not find any matches!
		+

	'''Notes:'''		'''Notes:'''

@@ Line 62: / Line 62: @@
 Warning: If you see squares, it can also be a font problem! This happen if you're using a font which does not define some characters you are using.
-=== A Mac:Central Europe text decoded as UTF-8 ===
+=== A Mac:Central Europe text decoded as ANSI ===
 === Using a font that does not include the characters you want ===

@@ Line 45: / Line 45: @@
 The two most common encoding are ISO-8859-1 and UTF-8
-=== 1. An UTF-8 text decoded as ISO-8859-1 (or ISO-8859-*) ===
+=== An UTF-8 text decoded as ISO-8859-1 (or ISO-8859-*) ===
 You will see two or three characters for non-ASCII ones, usually Ã and ©.

@@ Line 64: / Line 64: @@
 === A Mac:Central Europe text decoded as UTF-8 ===
-=== 3. Using a font that does not include the characters you want ===
+=== Using a font that does not include the characters you want ===
 If you use accented Greek characters, for example, but do not use a Unicode font that supports precomposed Greek characters (like @Arial Unicode MS) accented characters will appear as empty rectangles in your text. In BibleCS you can change the font in Options/Preferences/Display.

@@ Line 52: / Line 52: @@
   Et Dieu vit la lumiÃ¨re, qu'elle Ã©tait bonne; et Dieu sÃ©para la lumiÃ¨re d'avec les tÃ©nÃ¨bres.
-=== 2. An ISO-8859-1 text decoded as UTF-8 ===
+=== An ISO-8859-1 text decoded as UTF-8 ===
 The UTF-8 decoder will find an invalid multi-byte character, it can report an error, or put question marks or squares instead:
@@ Line 61: / Line 61: @@
 Warning: If you see squares, it can also be a font problem! This happen if you're using a font which does not define some characters you are using.
 === 3. Using a font that does not include the characters you want ===

@@ Line 41: / Line 41: @@
 == Common mistakes ==
-If you use a wrong encoding, some characters will not be displayed correctly as the computer decoded it using incorrect rules.
+If you use a wrong encoding, some characters will be displayed [http://en.wikipedia.org/wiki/Mojibake incorrectly] as the computer decoded it using incorrect rules.
 The two most common encoding are ISO-8859-1 and UTF-8

@@ Line 31: / Line 31: @@
 Now, Code-point are under 2^32, so we can represent every character with a 4 bytes word, (a 32 bit word). This is called [http://en.wikipedia.org/wiki/UTF-32/UCS-4 UCS4], or UTF-32.
-For an English text, because every character is encoded with 4 bytes, the size if 4 time bigger than the same text in ASCII encoding.
+For an English text, because every character is encoded with 4 bytes, the size is 4 time bigger than the same text in ASCII encoding.
 == Variable byte length encoding ==

← Older revision		Revision as of 16:43, 18 June 2015
Line 102:		Line 102:

	* [http://www.unicode.org/reports/tr15/ Unicode Normalization Forms] – Unicode Standard Annex #15		* [http://www.unicode.org/reports/tr15/ Unicode Normalization Forms] – Unicode Standard Annex #15
		+
		+	* [http://unicode.org/versions/Unicode8.0.0/ Unicode 8.0.0] – released 2015-06-17

	[[Category:Unicode]]		[[Category:Unicode]]

@@ Line 24: / Line 24: @@
 [http://en.wikipedia.org/wiki/Unicode Unicode] is an industry standard where every character in any real language, past or present (now over 100,000 characters), is having a different numeric value called a code-point. This code-point can be recorded in many ways. The first 128 code-points are the same values used in the ASCII encoding.
-Did you remember, Unicode is not an encoding? Ok, in fact, Unicode also defines many different encodings to store code-point.
+Did you remember, Unicode is not an encoding? Ok, in fact, Unicode also defines many different encodings to store code-points.
 == Multi-bytes encoding ==