Remove Duplicate Lines

Removing duplicate lines cleanly

Deduplicating a list means keeping one copy of each distinct line and dropping the rest. It sounds trivial until you hit the edge cases: are two lines that differ only by a trailing space duplicates? Does leading indentation count? Is Apple the same as apple? A good dedupe step lets you decide, usually with options to trim whitespace and fold case before comparing.

When it earns its keep

Typical uses are cleaning an email export before an import, collapsing a log of repeated error lines, merging two keyword lists, or tidying a list of IDs pasted from a spreadsheet. It pairs well with sorting, since a sorted-then-deduped list reads predictably, but order-preserving dedupe matters when the first occurrence should win.

Example

Input          Order-preserving   Sorted + unique
banana         banana             apple
apple          apple              banana
banana         cherry             cherry
cherry

Order-preserving keeps the first time each value appeared. Sorting first groups equal lines together, which is how command-line sort | uniq works and why uniq alone only removes adjacent repeats.

Common pitfalls

Invisible differences cause the most confusion. A line ending in CRLF is not equal to the same text ending in LF, so Windows and Unix line endings can leave apparent duplicates behind. Trailing spaces and a stray tab do the same. Decide up front whether comparison is exact or normalized.

FAQ

Does it keep the first or last occurrence? Order-preserving dedupe keeps the first by convention, so later repeats are the ones discarded.

Why do two lines that look identical survive? One almost certainly carries hidden whitespace or a different line ending. Clean the text with a whitespace cleaner first, then dedupe.

Can I sort at the same time? Yes, though sorting is a separate decision. If order does not matter, sorting makes the result easier to scan.