RCP: 0.7.5: Fix concordance export memory usage
The current Concordance export loads all the concordance Lines in memory before writing them into the CSV file which leads to memory exhaustion for voluminous concordance results.
Solution
A solution is to write the concordance lines per packet, letting early packets to be garbage collected to keep memory consumption level.
2 development steps :
A- Create a macro (to provide a rapid answer)
B- Fix Concordance.toTxt(…) in sources and produce an update
Recette macro (FR)
- Télécharger l’archive de la macro (pièce jointe)
- copier le dossier “export” que l’archive contient dans le dossier des macros de TXM (
``` (et non “macros” !)).
\- faire une concordance de “\[\]” sur BROWN
\- lancer la macro sur la concordance
- vérifier que le nombre de lignes (wc -l) est le nombre de résultat + 1
(l’entête)
Recette maj (FR)
----------------
\- …
\- faire une concordance de “\[\]” sur BROWN
\- lancer la macro sur la concordance
- vérifier que le nombre de lignes (wc -l) est le nombre de résultat + 1
(l’entête)
*(from redmine: issue id 763, created on 2014/04/24 by Matthieu Decorde)*Activity
changed milestone to %TXM 0.7.6
Packet size will be set to 5000.
R code to visualize the experimental export time graph (in milliseconds) in terms of packet size (line), for a concordance of 900k lines on a standard Linux workstation:
size <- c(10,100,1000,5000,10000,50000,100000) time <- c(57152,50090,49592,49525,50394,53320,58106) plot(size, time,type="p")(from redmine: written on 2014/04/24 by Matthieu Decorde)
assigned to @mdecorde
