Character Encoding Guide

Character Encoding Development History

Character encoding is the rule for mapping characters to numbers, forming the foundation of computer text processing. Encoding development has gone through several important stages: ASCII (1963, 7-bit encoding, 128 characters) laid the foundation for English character encoding; Extended ASCII (8-bit encoding, 256 characters) added Western European characters; GB2312 (1980, Chinese national standard) supports 6,763 Chinese characters; GBK (1995) extended GB2312, supporting 21,886 Chinese characters; Unicode (1991) aimed to unify all writing systems globally.

Unicode is currently the most universal character set, containing over 140,000 characters covering 159 writing systems. UTF-8 is Unicode's variable-length encoding, using 1-4 bytes to encode a character, fully compatible with ASCII, and has become the internet's mainstream encoding. UTF-16 uses 2 or 4 bytes and is the internal encoding for Java and JavaScript. UTF-32 uses a fixed 4 bytes, which is convenient to process but wastes space.

Encoding Conversion and Handling

// JavaScript character encoding handling

// TextEncoder/TextDecoder for encoding handling
const encoder = new TextEncoder();  // Default UTF-8
const decoder = new TextDecoder();  // Default UTF-8

// String to Uint8Array
const utf8Bytes = encoder.encode('Hello World');
console.log(utf8Bytes);  // Uint8Array [72, 101, 108, ...]

// Uint8Array to string
const text = decoder.decode(utf8Bytes);
console.log(text);  // 'Hello World'

// Detect encoding
function detectEncoding(buffer) {
  const bytes = new Uint8Array(buffer);

  // Check BOM
  if (bytes[0] === 0xEF && bytes[1] === 0xBB && bytes[2] === 0xBF) {
    return 'UTF-8';
  }
  if (bytes[0] === 0xFF && bytes[1] === 0xFE) {
    return 'UTF-16LE';
  }
  if (bytes[0] === 0xFE && bytes[1] === 0xFF) {
    return 'UTF-16BE';
  }

  return 'UNKNOWN';
}

Troubleshooting Garbled Text

Garbled text is a common issue caused by encoding inconsistency. Troubleshooting steps: first confirm the file's actual encoding (using a hex editor to check BOM or byte patterns); then check if the encoding specified when reading is correct; finally check the encoding settings when outputting. Best practices to prevent garbled text: always use UTF-8 encoding; declare charset in HTML: <meta charset="UTF-8">; specify UTF-8 encoding in database connections; explicitly specify encoding for file read/write operations.

In web development, ensuring consistent encoding across all stages is crucial. HTTP response headers should include Content-Type: text/html; charset=utf-8. Database tables and fields should use utf8mb4 encoding (supporting complete Unicode, including emoji). JSON data returned by APIs should also explicitly specify encoding.

Recommended Tools

Notepad++ is a text editor supporting multiple encodings. iconv is a command-line encoding conversion tool. chardet is a Python character encoding detection library. Charset Detector is an online encoding detection tool.