Common Encoding Mistakes and How to Fix Them: A Developer's Guide

Encoding is one of those things that developers don't think about until it breaks — and when it breaks, it can break in spectacular, confusing ways. I've spent days debugging encoding issues that boiled down to a simple misunderstanding.

Let me start with a story from a previous job. We had an e-commerce site where product names with accents (like "Café") would sometimes show up as "Café" in the email receipts, but look fine on the website. It took us weeks to track down the issue: the product database was using UTF-8, the email system was expecting Latin-1, and somewhere in between, the encoding was being misdetected and converted incorrectly.

Here's the core principle: Encoding is how we convert between human-readable text and the bytes that computers store and transmit. When both sides agree on the encoding, everything works. When they don't — or when you encode multiple times by accident — you get garbage.

The good news is that most encoding mistakes are predictable and avoidable once you know what to look for. This guide covers the mistakes I see most often, how to recognize them, and how to fix them.

Double encoding happens when you encode data that's already been encoded. It's surprisingly easy to do, especially in complex systems where multiple layers touch the data.

Let's look at a URL encoding example. Start with the text: "Hello world"

- Encode once: "Hello%20world" (the space becomes %20)

- Encode again: "Hello%2520world" (the % itself gets encoded to %25)

Notice that the `%` character becomes `%25` on the second pass. That's the telltale sign of double encoding: seeing `%25` where you'd expect just `%`.

Another example with HTML entities:

- Original: "Smith & Wesson"

- Encode once: "Smith & Wesson"

- Encode again: "Smith & Wesson"

When this renders, users will see "Smith & Wesson" instead of "Smith & Wesson" — the browser only decodes once.

**Real-world example:** I once saw an API where the frontend would URL-encode a search query, then the API gateway would encode it again before passing it to the backend. Searching for "A&W" became "A%2526W", and the backend couldn't find any results. It took hours to debug because the logs showed the encoded version, and we didn't notice the `%25` at first.

**How to fix double encoding:**

- Map out where encoding happens in your stack. Document: "This layer encodes the URL parameters, this other layer expects already-encoded data", etc.

- Only encode once, at the last responsible moment. Don't encode data early and pass it around — encode it right before you send it to the system that needs it encoded.

- If you receive data that might be double-encoded, you can sometimes decode once, check if it still looks encoded, and decode again if needed. But this is a workaround, not a solution — fix the root cause instead.

- Use types to your advantage. In TypeScript, you could have a `UrlEncodedString` type that can only be created by a function that actually does the encoding, preventing accidental re-encoding.

This is another classic: you have a URL with parameters, and you encode the entire thing instead of just the parameter values. The result is a mess.

**What NOT to do:** `https%3A%2F%2Fexample.com%2Fpage%3Fid%3D123%26category%3Dshoes`

That's the entire URL encoded, including the `https://`, the path, the `?`, and the `&` separators. This won't work in a browser because the browser can't tell where the scheme ends and the host begins.

**What TO do:** Encode only the values that might contain special characters:

`https://example.com/page?id=123&category=shoes%20%26%20socks`

Notice that the `?`, `&`, and `=` are preserved as separators. Only the actual values get encoded. In this example, the category is "shoes & socks", so the space becomes `%20` and the ampersand becomes `%26`.

**Why this happens:** I've seen this when developers copy a URL, paste it into an encoding tool, and encode the whole thing. Or when they use a function that encodes the entire URL instead of just the parameters.

**How to do it correctly in JavaScript:**

// Don't encode the whole URL!
const bad = encodeURIComponent('https://example.com/page?id=123&category=shoes & socks');

// Do encode just the values, then build the URL
const baseUrl = 'https://example.com/page';
const params = new URLSearchParams({
  id: '123',
  category: 'shoes & socks' // This value will be automatically encoded
});
const good = `${baseUrl}?${params}`;
// Result: 'https://example.com/page?id=123&category=shoes+%26+socks'

// Or manually encode just the values
const category = 'shoes & socks';
const manual = `https://example.com/page?id=123&category=${encodeURIComponent(category)}`;

The `URLSearchParams` API is your friend here — it handles encoding the values correctly automatically.

**Pro tip:** Always build URLs with a library or built-in API instead of string concatenation. It's too easy to forget to encode something, or encode the wrong part.

UTF-8 should be the default encoding for everything these days, but I still see issues with it constantly. Emoji, accented characters, and non-Latin scripts are the canary in the coal mine — if those break, you have an encoding problem.

**Common symptoms:**

- The replacement character: `�` instead of the expected character. This means the decoder encountered bytes that don't make sense in the expected encoding.

- Mojibake (garbled text): "Café" instead of "Café", or "こんにちは" instead of "こんにちは". This happens when you decode UTF-8 as Latin-1 (or vice versa).

- Missing characters, or boxes with question marks or hex codes inside.

**Why this happens:**

- Your database isn't using UTF-8 (or is using "utf8" in MySQL, which isn't actually full UTF-8 — use `utf8mb4` instead).

- You're reading a file with the wrong encoding (e.g., reading a UTF-8 file as Latin-1).

- You're not specifying the encoding when converting between bytes and strings.

- Different layers of your stack are using different encodings.

**Real-world story:** I once worked on a messaging app where emoji would work on the web but not on iOS. It turned out the backend was storing data as Latin-1, which can't represent emoji. When an iOS user sent an emoji, the backend would replace it with `?`. The fix was to migrate the entire database to UTF-8, which was not a fun weekend project.

**How to fix it:**

- **Use UTF-8 everywhere.** I can't stress this enough. Your database, your API, your HTML, your files — everything should be UTF-8.

- **Explicitly specify UTF-8** when converting between bytes and strings. Don't rely on defaults, because they vary by system and language.

- **In HTML:** Include `<meta charset="utf-8">` in the `<head>` of every page.

- **In databases:** Use the UTF-8 character set. For MySQL, that means `utf8mb4` (which supports emoji, unlike the old `utf8`).

- **In files:** Save your source files as UTF-8 (most editors do this by default now, but check).

- **In APIs:** Set the `Content-Type` header to include the charset: `Content-Type: application/json; charset=utf-8`.

Here's what not to do: Don't try to "clean" the text by removing non-ASCII characters. That's throwing away data. Just use UTF-8 properly instead.

HTML entities and URL encoding are two different systems for two different contexts. They look somewhat similar (they both use special characters), but they serve different purposes and you can't mix them up.

**HTML entities** are used in HTML documents to represent characters that have special meaning in HTML, or that can't be easily typed. They look like `&amp;`, `&lt;`, `&gt;`, `&quot;`, or `&#x2764;`.

Examples:

- `&` becomes `&amp;`

- `<` becomes `&lt;`

- `>` becomes `&gt;`

- `"` becomes `&quot;`

- `'` becomes `&#39;` or `&apos;`

HTML entities are used when you want to display these characters in HTML without them being interpreted as HTML markup.

**URL encoding (percent-encoding)** is used in URLs to represent characters that are not allowed or have special meaning in URLs. They look like `%26`, `%3C`, `%3E`, `%22`.

Examples:

- `&` becomes `%26`

- `<` becomes `%3C`

- `>` becomes `%3E`

- `"` becomes `%22`

- Space becomes `%20` or `+`

**The confusion:** Both systems encode the same characters but in different ways. If you use HTML entities in a URL, or URL encoding in HTML, it won't work correctly.

**Real-world example:** I once saw a link like `<a href="search?q=cat%26dog">` — that's correct URL encoding. But then someone changed it to `<a href="search?q=cat&amp;dog">` — that's HTML entities in the URL, which is wrong. In the HTML, `&amp;` becomes `&`, so the browser sees `q=cat&dog` as two separate parameters, not one parameter with the value "cat&dog".

**How to remember:**

- Inside HTML content or attributes: Use HTML entities

- In URLs: Use URL encoding (percent-encoding)

- When building HTML that contains URLs: First URL-encode the parameters, then HTML-encode the entire URL if necessary (though usually URLs don't contain HTML-special characters except `&`, which is already handled by URL encoding)

And yes, sometimes you need to encode twice in this specific order: URL-encode first, then HTML-encode — but only if you're putting a URL inside HTML.

This mistake is so common and so dangerous that it deserves its own section. Base64 is encoding, not encryption. It looks like gibberish, but it's trivial to decode — you can do it with our Base64 tool in 2 seconds.

**What I've seen:**

- Storing API keys in Base64 in config files

- Putting passwords in Base64 in query parameters

- Sending sensitive data in Base64 over unencrypted HTTP

- A startup that stored credit card numbers in Base64 in their database (they got hacked, and it was bad)

**Why this is bad:** Anyone who gets the Base64 string can decode it immediately. Base64 doesn't protect data at all — it just changes its representation.

**The difference:**

- **Encoding:** Changes the representation of data, not its confidentiality. Anyone can decode it. Base64, URL encoding, HTML entities — all encoding.

- **Encryption:** Transforms data so that only authorized parties can read it. Requires a key. AES, RSA, etc.

- **Hashing:** One-way transformation — you can't get the original data back. Used for passwords, checksums. SHA-256, bcrypt, etc.

**What to do instead:**

- For storing secrets: Use a secret management system like HashiCorp Vault, AWS Secrets Manager, or Azure Key Vault.

- For storing passwords: Use a slow, salted hashing algorithm like bcrypt, Argon2, or PBKDF2. Never store passwords in plaintext or Base64.

- For sending sensitive data: Use HTTPS (TLS). That's what it's for.

- For encrypting data: Use a proper encryption algorithm with a key management system.

I once audited a system where the developers thought Base64 was "good enough" because "who would know to decode it?" That's security through obscurity, and it never works. Always use real security for sensitive data.

Unicode has several escape formats, and they're not interchangeable. I've seen many bugs where someone uses the wrong escape sequence for the context.

**Common Unicode escape formats:**

- **JavaScript/JSON:** `\uXXXX` (four hex digits), e.g., `\u00E9` for `é`, `\u2764` for `❤`. For characters outside the Basic Multilingual Plane, you need surrogate pairs: `\uD83D\uDE00` for `😀`

- **JSON6/ES2015+:** `\u{XXXXX}` (one to six hex digits in curly braces), e.g., `\u{1F600}` for `😀`

- **HTML:** `&#xXXXX;` or `&#DDDDD;`, e.g., `&#xE9;` or `&#233;` for `é`, `&#x2764;` for `❤`

- **CSS:** `\XXXX ` (four hex digits followed by a space), e.g., `\00E9 ` for `é`

- **URL:** `%XX` for bytes (UTF-8 encoded), e.g., `%C3%A9` for `é`

**The mistake:** Using a JavaScript escape in HTML, or an HTML escape in JSON, etc. They don't work the same way!

**Example:** If you put `\u00E9` in HTML, it will literally show up as "\u00E9", not as "é". You need `&#xE9;` or just the actual character `é` in HTML.

**What to do:**

- In most cases, just use the actual Unicode character directly. Modern tools and systems handle this fine.

- If you need to escape (e.g., for control characters), use the escape format appropriate for the context.

- Our Unicode Escape tool can convert between text and these escape formats for you.

**Real-world example:** I once saw a translation system where the translators were putting JavaScript escapes (`\u00E9`) into the translation files, but the files were YAML, not JavaScript. The result was the literal escape sequences showing up in the UI instead of the accented characters.

This is the most dangerous mistake: forgetting to encode data when it moves between contexts. This is how SQL injection, XSS (cross-site scripting), and other injection attacks happen.

**XSS example:**

You have a comment system, and a user writes: `<script>stealCookies()</script>`

If you put that directly into HTML without encoding it, the browser will execute the script. That's XSS.

If you HTML-encode it first: `&lt;script&gt;stealCookies()&lt;/script&gt;`, the browser will display it as text instead of executing it.

**SQL injection example:**

You build a query like: `const query = `SELECT * FROM users WHERE id = ${userId}`;`

If a malicious user sets `userId` to `123 OR 1=1`, you get: `SELECT * FROM users WHERE id = 123 OR 1=1` — which returns all users.

Instead, use parameterized queries: the database library handles encoding the values correctly.

**The rule:** Always encode/escape data when it moves from one context to another. The type of encoding depends on the destination context:

- From user input to HTML: HTML-encode it

- From user input to SQL: Use parameterized queries

- From user input to URL: URL-encode it

- From user input to shell commands: Avoid shell commands with user input entirely, but if you must, escape appropriately

This isn't just about encoding for display — it's about security. Injection attacks are still in the OWASP Top 10 because people still forget to do this.

When I'm debugging an encoding issue, I go through these questions systematically. It saves me hours of guesswork.

**1. What's the original data?** Get as close to the source as possible. If it's from a database, look at the raw bytes in the database. If it's from a file, look at the file in a hex editor if you need to.

**2. What encoding is it in right now?** Don't guess — check. Look at the raw bytes if you can. In JavaScript, `TextEncoder` can show you the UTF-8 bytes. In Node.js, `Buffer.from()` is helpful.

**3. What encoding should it be in?** What does the destination system expect? Is it UTF-8? Latin-1? Something else?

**4. How many times has it been encoded?** Look for the telltale signs: `%25` means double URL-encoding, `&amp;amp;` means double HTML-encoding.

**5. Is the correct part being encoded?** Are you encoding just the values, or the whole URL? Are you encoding the data, not the syntax?

**6. Can you decode it back to the original?** A good sanity check: encode, then decode — do you get back what you started with?

**7. Are you using the right type of encoding for the context?** HTML entities in HTML, URL encoding in URLs, etc.

**8. Are you handling Unicode correctly?** Are you using UTF-8 everywhere? Are you specifying the encoding explicitly when converting between bytes and strings?

**9. Is this a security issue?** Could this lead to XSS, SQL injection, or another vulnerability?

And my final piece of advice: When in doubt, look at the raw bytes. Strings can be misleading, but bytes don't lie. A hex editor or a function that shows you the actual byte values can clarify what's really going on.