Improve memory manage of gridtab csv import

hiep, perhaps you are not using native sequence ? for big data set, that should make a big difference.

should turn off change log for big import and I think the update to recent items could be an issue too.

thanks for comment. finally i import by sql with id = current + row_number()
after that update seq, very fast, about 3 minute.
about long time transaction, i read discuss here and agree with it.

about tab csv import. it can slow but shouldn't take a lot of memory
not yet take time to discover, but i guess by it keep one install of window to import data, so end up it like you load 500.000 record on this window.
so it should refresh window instance after like 100 master record (that should on parameter).

, 2pack is designed to do all the work within a single transaction, so loading 500.000 records with 2pack can lead to a big long trx - I think is not the intention of 2pack to load such big data.

About CSV import - it can work within a single transaction or transaction per line depending on the configuration of the CSV file - for better results in a big load case like you try the transaction must be per line.

I think CSV import is slow and memory-hungry because it was designed to emulate a user typing in the window, so, it uses all the UI-related structures. That has a big advantage and is that callouts are executed as if the user is typing in the window, but cannot give the same speed and memory than a direct-model importer (not using the UI).

We have annotated that as a "known-issue" for CSV importer and recommendation is to split the file in manageable sizes. Or, for big data, use the old importers when available, in the case of product price import there is an old non-UI importer that can be used.

But, of course, any improvement in memory/speed for the CSV importer that doesn't break the UI-emulation functionality is welcome.

2Pack also got issue with huge record.
i try to use 2Pack for about 500.000 product price
it take long time (about 7h) and end with exception transaction timeout
i think overload stay at get ID of table and save change log