Working with UTF-16 Encoded Files in Linux

Issue: Working with UTF-16 Encoded Files in Linux

Problem:

When trying to process a binary/encoded file (app1_disk_old.log), tools like grep, strings, and less don't return expected results. The file is identified as UTF-16 encoding (little-endian) with CRLF line terminators, causing issues when searching or processing with typical text-processing tools.

Solution:

To resolve the issue, the file needs to be converted from UTF-16 to UTF-8 encoding.

Steps to Fix:

  1. Convert the File from UTF-16 to UTF-8: Use the iconv tool to convert the file encoding to UTF-8:
 iconv -f UTF-16 -t UTF-8 app1_disk_old.log > app1_disk_old_utf8.log

The converted file will be saved as app1_disk_old_utf8.log.

file app1_disk_old_utf8.log

Ensure it now shows UTF-8.

Search the Converted File: Now that the file is in UTF-8 format, use standard text-processing tools like grep to search for specific terms (e.g., "app1"):

grep "app1" app1_disk_old_utf8.log
strings app1_disk_old_utf8.log | grep "app1"
A.M. Rinas

Written by A.M. Rinas

Full-Stack Software Engineer working on web applications, DevOps automation, and databases. More about me on the about page, or say hi through the contact page and LinkedIn.

← Back to all articles