Near-Duplicate File Finder

Near-duplicate file detection is the process of finding files that are similar but not identical - unlike exact duplicate detection which only finds files with identical content. Near-duplicates share significant content but may differ in metadata, formatting, compression, edits, or re-encoding. Our tool uses a fuzzy hashing algorithm inspired by ssdeep (Context Triggered Piecewise Hashing or CTPH) that divides files into content-based chunks and compares their chunk signatures to compute a similarity score. This approach detects files that exact hash comparison would miss, making it ideal for finding related documents, similar images, versioned files, and modified copies.

1. Content-Based Chunking:The file is read as bytes and a rolling hash (similar to Adler-32) slides through the data. When the hash value matches a specific trigger pattern, a chunk boundary is created. This means chunks align with the file's natural content structure rather than fixed positions - making the algorithm robust against insertions and deletions.
2. Chunk Hashing: Each content chunk is hashed using SHA-256 via the Web Crypto API. The resulting hash signatures capture the content of each chunk. Identical chunks across different files produce identical hashes, allowing the algorithm to detect shared content even when the files differ in other parts.
3. Similarity Computation: Files are compared by computing three metrics: exact chunk match ratio (how many chunks have identical hashes), longest common subsequence of chunks (how much of the chunk order is preserved), and file size ratio. These are weighted and combined into a final similarity score from 0% to 100%.
4. Batch Comparison: When multiple files are uploaded, every unique pair is compared automatically. Results are sorted by similarity descending and categorized into five levels: Identical, Near-Duplicate, Similar, Somewhat Similar, and Different.

• Metadata Changes: Files with the same content but different metadata (EXIF data in images, document properties, file timestamps) are detected as near-duplicates since the content chunks remain largely the same.
• Formatting Differences: Documents reformatted with different fonts, spacing, or layout will still share most content chunks, making them detectable as near-duplicates even when the visual appearance differs.
• Minor Edits: Files with small additions, deletions, or modifications (like updated paragraphs, corrected typos, or added comments) will share the majority of unchanged content chunks, resulting in high similarity scores.
• Re-encoding/Re-compression: Images re-saved at different quality levels or files re-compressed with different settings will share underlying content structure, though chunk boundaries may shift, resulting in moderate-to-high similarity scores.

Our Near-Duplicate File Finder processes everything locally in your browser. Files are read using the FileReader API and hashed using the Web Crypto API. No data is ever uploaded to any server - your files stay completely private on your device. The tool is 100% free with no signup, no account, and no usage limits.

Important limitations: The algorithm is designed for content-based similarity detection. Files that are semantically similar but have completely different binary content (e.g., the same text saved in PDF vs. DOCX format) will not be detected as near-duplicates. Very small files (under 64 bytes) may produce insufficient chunks for meaningful comparison. Processing time increases with the number of files since every pair must be compared. For very large sets (50+ files), consider processing in smaller batches.