improve CSV Import performance

Description

The CSV importer can be used to import quite huge data into iDempiere. Since it can be used in even more cases. We had a test run with a list of prices & conditions with 102.000 entries. It took about one hour which is a lot. I want to improve that.

One thing is that we found some places in the importer code where java objects are created inside loops.

Another thing is that we can use asynchronous commits (http://www.postgresql.org/docs/9.1/static/wal-async-commit.html). We tested a small patch and got a speed gain of 30% (with 5000 records). We did this with activated debug messages so the end result should be even better.

This page in the PostgreSQL wiki suggest to switch off autovacuum. I did not test this yet.

Environment

None

Activity

Show:
Carlos Ruiz
May 20, 2015, 1:30 PM

Assignee

Diego Ruiz

Reporter

Thomas Bayen

Labels

Tested By

Diego Ruiz
Thomas Bayen

Priority

Major
Configure