Distributed cache kills performance

Description

Reviewing a very bad performance on an installation using ASP (SaaS) module, noticed that the caches distributed using hazelcast are killing the peformance.

The cause is that every call to the cache is returning a different object, and the cache of ASP fields is implemented as an array of the object - trusting that every call returns the same object.

So, in practice every call to MClient.get is returning a different MClient object and forcing a new expensive query in MClient.isDisplayField

Two solutions analyzed:

1 - make the cache of MClient non distributed

2 - implement near-cache in hazelcast.
See http://docs.hazelcast.org/docs/latest-dev/manual/html-single/index.html#near-cache

Regards,

Carlos Ruiz

Environment

None

Activity

Show:
Carlos Ruiz
November 28, 2018, 3:57 PM

But also please consider the following - the hazelcast-template.xml is intended as a template that works fine out-of-the-box -but every installation can modify the hazelcast.xml on the IDEMPIERE_HOME folder for its own needs - indeed when setting up hazelcast for multi-server (example AWS) it is required to modify this file.

Hiep Lq
November 28, 2018, 4:59 PM

Hi mr , hazelcast do distribute cache like bellow step.

on binary format:
put instance_1 action:
1. serialization instance_1 to binary, and save binary to store (now if you change something of instance_1 it don't effect binary on store)

get action:
1. take binary of object from store. and do deserialization become instance_2
2. return instance_2 to caller so if you change something on instance_2 it don't effect to binary on store

continue get action:
1. take binary of object from store. and do deserialization become instance_3
2. return instance_3 to caller so if you change something on instance_3 it don't effect to binary on store and instance_2

my fix will update binary after you change instance_2, so next time when you do get action, you take instance_3.
it's difference to instance_2. but it already have m_fieldAccess, so don't run query again

Hiep Lq
November 28, 2018, 5:01 PM

the point of my fix is to update object on store, so ever hazelcast.xml update format from binary to object it still work

Hiep Lq
November 28, 2018, 5:06 PM

when idempiere run as standalone (don't active any join method like default) i suggest to don't use distribute cache to save cost of deserialization process

Carlos Ruiz
December 23, 2018, 4:26 PM

Hi , I tested your patch in a ASP implementation.

Effectively your patch avoids the expensive query being executed several times.

However the performance of the system is a lot slower with BINARY than with the near cache set in OBJECT format.

The cache still keeps creating one object on each call, and the number of calls to MClient.get is really big.

Regards,

Carlos Ruiz

Fixed

Assignee

Carlos Ruiz

Reporter

Carlos Ruiz

Labels

None

Tested By

None

Affects versions

Priority

Major
Configure