Some of the file names in /usr/local/share/corp/orig contain non-utf8 characters or are otherwise troublesome. Some of the characters are harmful when passed to shell, like parentheses, like in file:
The non-uft8 chars in filenames can be spotted in http://www.divvun.no/doc/lang/corp/corpus-sme.html
Not all of them cause errors in coversion.
- The script convert2xml.pl is fixed so that the files with troublesome characters are not accepted to the conversion (quotation marks, parentheses, &, etc.). The directories under sme/gtbound and sme/gtfree are fixed so that the broken files are removed and reconverted so that only valid filenames are accepted. I'll write some documentation of the accepted filenames (general unix style with sámi chars is preferred).
- The filenames under sme/orig are moved to NFC-format to ease up browsing and editing, the converted files in sme/gtbound and sme/gtfree end up to NFC as well.
- Some of the errors in the filenames were caused by the module Decode.pm, which still had the line
"use encoding utf-8" which converted the utf8 -characters to utf8 again. Some of the errors were caused by missing quotation marks around filenames in convert2xml.pl These are now fixed.
The files in smj are fixed as well, but other languages are not. I'll do the fix if needed.