View ProjeQtOr On SourceForge.net
ProjeQtOr - Project Management Tool
Support us on Capterra
OIN - Open Invention Network
ProjeQtOr free project management software - history grow too fast when using projeqtor with daily(or more) cron import - ProjeQtOr
 
 

history grow too fast when using projeqtor with daily(or more) cron import

More
27 Sep 2022 17:16 #1 by loicr
history grow too fast when using projeqtor with daily(or more) cron import 

We are importing a lot of datas 3 times per days ( bill / customer / supplier / expense / order... ) but our systeme is not able to send only modifications so we import in mass and projeqtor cron is makihng update or not ...

Since several days, we see that projeqtor is really slow when we put project details on screen, first all all we were thinking of calculation for invoice / order / expense ... but yesterday it was so ugly that we try to find the trouble .... 
History table was more than 600 000 lines and 400mo .... !!!
We discover that for each line of import even when existing, projeqtor write in history for calculation update / margin update ... cost  !!!!!!! 
With 3 batchs imports per days ( around 20.000 lines of order / invoice / ... ) for updating from CEGID ... situation is not realistic :)

Could it be possible to desactivate history for importation, with an option for exemple or with other solution ?
Now we create a script specific to erase datas history from import ... but it is really complex and other who want to synchronise will have the same trouble...

History from datas imported several times a day is not pertinent for us

Do you have a solution to avoid below script ?
............................
mysql --host=XXXXXXXXXXXXXXXXXXXXXX --binary-mode=1 --database=projeqtor -e "DELETE FROM history WHERE reftype='ProjectExpense'"
mysql --host=XXXXXXXXXXXXXXXXXXXXXX --binary-mode=1 --database=projeqtor -e "DELETE FROM history WHERE reftype='ProviderOrder'"
mysql --host=XXXXXXXXXXXXXXXXXXXXXX --binary-mode=1 --database=projeqtor -e "DELETE FROM history WHERE reftype='Command'"
mysql --host=XXXXXXXXXXXXXXXXXXXXXX --binary-mode=1 --database=projeqtor -e "DELETE FROM history WHERE reftype='ProviderBill'"
mysql --host=XXXXXXXXXXXXXXXXXXXXXX --binary-mode=1 --database=projeqtor -e "DELETE FROM history WHERE reftype='Provider'"
mysql --host=XXXXXXXXXXXXXXXXXXXXXX --binary-mode=1 --database=projeqtor -e "DELETE FROM history WHERE reftype='ImportLog'"
mysql --host=XXXXXXXXXXXXXXXXXXXXXX --binary-mode=1 --database=projeqtor -e "DELETE FROM history WHERE reftype='bill'"
mysql --host=XXXXXXXXXXXXXXXXXXXXXX --binary-mode=1 --database=projeqtor -e "DELETE FROM history WHERE reftype='Expense'"
mysql --host=XXXXXXXXXXXXXXXXXXXXXX --binary-mode=1 --database=projeqtor -e "DELETE FROM history WHERE reftype='ProjectPlanningElement' and colName='billSum'"
mysql --host=XXXXXXXXXXXXXXXXXXXXXX --binary-mode=1 --database=projeqtor -e "DELETE FROM history WHERE reftype='ProjectPlanningElement' and colName='plannedCost'"
mysql --host=XXXXXXXXXXXXXXXXXXXXXX --binary-mode=1 --database=projeqtor -e "DELETE FROM history WHERE reftype='ProjectPlanningElement' and colName='realCost'"
mysql --host=XXXXXXXXXXXXXXXXXXXXXX --binary-mode=1 --database=projeqtor -e "DELETE FROM history WHERE reftype='ProjectPlanningElement' and colName='expenseAssignedAmount'"
mysql --host=XXXXXXXXXXXXXXXXXXXXXX --binary-mode=1 --database=projeqtor -e "DELETE FROM history WHERE reftype='ProjectPlanningElement' and colName='expenseLeftAmount'"
mysql --host=XXXXXXXXXXXXXXXXXXXXXX --binary-mode=1 --database=projeqtor -e "DELETE FROM history WHERE reftype='ProjectPlanningElement' and colName='expensePlannedAmount'"
mysql --host=XXXXXXXXXXXXXXXXXXXXXX --binary-mode=1 --database=projeqtor -e "DELETE FROM history WHERE reftype='ProjectPlanningElement' and colName='totalAssignedCost'"
mysql --host=XXXXXXXXXXXXXXXXXXXXXX --binary-mode=1 --database=projeqtor -e "DELETE FROM history WHERE reftype='ProjectPlanningElement' and colName='totalLeftCost'"
mysql --host=XXXXXXXXXXXXXXXXXXXXXX --binary-mode=1 --database=projeqtor -e "DELETE FROM history WHERE reftype='ProjectPlanningElement' and colName='totalPlannedCost'"
mysql --host=XXXXXXXXXXXXXXXXXXXXXX --binary-mode=1 --database=projeqtor -e "DELETE FROM history WHERE reftype='ProjectPlanningElement' and colName='totalRealCost'"
mysql --host=XXXXXXXXXXXXXXXXXXXXXX --binary-mode=1 --database=projeqtor -e "DELETE FROM history WHERE reftype='ProjectPlanningElement' and colName='expenseRealAmount'"
mysql --host=XXXXXXXXXXXXXXXXXXXXXX --binary-mode=1 --database=projeqtor -e "DELETE FROM history WHERE reftype='ProjectPlanningElement' and colName='commandSum'"
mysql --host=XXXXXXXXXXXXXXXXXXXXXX --binary-mode=1 --database=projeqtor -e "DELETE FROM history WHERE reftype='ProjectPlanningElement' and colName='marginCost'"
mysql --host=XXXXXXXXXXXXXXXXXXXXXX --binary-mode=1 --database=projeqtor -e "DELETE FROM history WHERE reftype='ProjectPlanningElement' and colName='marginCostPct'"


Thanks for your support.
LR

Please Log in or Create an account to join the conversation.

More
28 Sep 2022 17:36 #2 by babynus

Since several days, we see that projeqtor is really slow when we put project details on screen

If this is due to history, just try and not display history systematically (activate the option to show "on request"

History table was more than 600 000 lines

That's not that much, I already saw 4 000 000, working fine

With 3 batchs imports per days ( around 20.000 lines of order / invoice / ... ) for updating from CEGID ... situation is not realistic

Possibly this is not an issue with projeqtor, but with what you try to do with it.

Could it be possible to desactivate history for importation, with an option for exemple or with other solution ?

This does not exist, and I'm not sure it is desirable feature.

Now we create a script specific to erase datas history from import ... but it is really complex and other who want to synchronise will have the same trouble...

I confirm that everyone who tries to import 20 000 changes 3 times per day will face same issue

Do you have a solution to avoid below script ?

Maybe just delete with criteria on idUser ?
I don't remind if idUser for automatic import is 1 or null

Babynus
Administrator of ProjeQtOr web site

Please Log in or Create an account to join the conversation.

More
29 Sep 2022 11:37 #3 by loicr
OK, i'll try to "not show" history to see if it is reacting better.
Otherwise i'll take in account your comment.

LR

Please Log in or Create an account to join the conversation.

Moderators: babynusprotion
Time to create page: 0.035 seconds

Cookies settings

×

Functional Cookies

Ce site utilise des cookies pour assurer son bon fonctionnement et ne peuvent pas être désactivés de nos systèmes. Nous ne les utilisons pas à des fins publicitaires. Si ces cookies sont bloqués, certaines parties du site ne pourront pas fonctionner.

Session

Please login to see yours activities!

Other cookies

Ce site web utilise un certain nombre de cookies pour gérer, par exemple, les sessions utilisateurs.