import, impossible to tokenize words written with point (.) characters inside

Given transcription principles using point characters inside words, for example the following TXT input where words are separated by space :

ḫr ḥm nỉ Ḥrw ‘nḫ-mst.pl nb.tỉ ‘nḫ-mst.pl nswt-bỉtỉ Ḫpr-kȝ-R‘

A) It is not possible to find correct XTZ or TXT import module parameters values to tokenize words with points inside.

Even when removing punctuations regex and point from sentence segmentation parameters.

Given impossibility to provide a import.xml file for example parameters, here is a screenshot of the parameters setting: import-txt-words-no-point.png

Here is the index of the “.\..” CQL : import-txt-words-no-point-words-with-points.png

B) Points are always rendered in editions with respect to default point formating rules of the current language

MD: when correctly tokenized the points rendering (in Edition and Concordance) is OK

See edition screenshot: import-txt-words-no-point-edition.png

(from redmine: issue id 3389, created on 2023/05/15 by Serge Heiden)

• Uploads:
  • import-txt-words-no-point.png
  • import-txt-words-no-point-words-with-points.png
  • import-txt-words-no-point-edition.png

added import type::bug + 1 deleted label

changed milestone to %TXM 0.8.4

The import parameters were re-initialized.

(from redmine: written on 2023/05/17 by Matthieu Decorde)

Index result when removing “.” from the tokenizer import parameters

word   Fréquence
‘   3
nḫ-mst.pl   2
ḥm  1
Ḫpr-kȝ-R    1
ḫr  1
Ḥrw 1
nb.tỉ   1
nỉ  1
nswt-bỉtỉ   1

Edition rendering :

ḫr ḥm nỉ Ḥrw ‘ nḫ-mst.pl nb.tỉ ‘ nḫ-mst.pl nswt-bỉtỉ Ḫpr-kȝ-R ‘

(from redmine: written on 2023/05/17 by Matthieu Decorde)

Correct token parameters usage has been verified on the test text sample (for ‘accented characters’ and ‘sentence end characters’).

(from redmine: written on 2023/05/22 by Serge Heiden)

closed

assigned to @mdecorde