Import, Tokenizer, re-tokenize words option

If enable the Tokenizer can retokenize words already wrapped with a element.

Enabled for the :

• XTZ import
• XML/w import
• transcriber

labels:

• re-tokenize pre-encoded words `flyover(Performs word segmentation within word encoding tags.)
  • re-segmenter lexicalement les mots pré-encodés `flyover(Réalise une segmentation en mots au sein des balises d’encodage de mots.)

(from redmine: issue id 3004, created on 2021/01/22 by Matthieu Decorde)

• Changesets:
  • Revision 3005 by Matthieu Decorde on 2021/01/29 08:34:11 +0100:

add the re-tokenize import parameter refs #3004

added import type::feature + 1 deleted label

changed milestone to %TXM 0.8.2 - 13NOV 1.0

ui is ready

(from redmine: written on 2021/01/29 by Matthieu Decorde)

tested with XTZ, XML/w and transcriber import modules

(from redmine: written on 2021/01/29 by Matthieu Decorde)

closed

assigned to @mdecorde