Run CLI action runFromCSV taking longer each time to import issues into jira

Description

We are using CSV Macro on a confluence page to import data into jira. While the csv macro works fine for smaller load but with larger files and consecutive user calls, the import gets slower and slower. We did a few test runs using a number of different users each uploading a same file of 500 issues from the confluence page.
It turns out that with each import run the execution time increase and it took a lot more time to execute the last few import runs. I have summarised the times below. We also observed high CPU usage during this time. Unfortunately we have not been able to figure out the root cause as yet and therefore would like you to share your thoughts as to why its taking longer when more calls are initiated to import data, even though the import runs sequentially and not in parallel.

Also find attached the snapshots of the high CPU and memory usage

Test Run Summary (times are reported based on the first and the last issue created based on the log file generated)

Users

Time Taken

Time taken to create One Issue

Note

User One

5 mins

0.58 sec

User Two

9 mins

1.04 sec

User Three

9 mins

1.04 sec

User Four

12 mins

1.38 sec

User Five

13 mins

1.48 sec

User Six

16 mins

1.69 sec

User Seven

22 mins

2.68 sec

User Eight

51 mins

6.18 sec

Not all issues were created for this run - 4 errors

User Nine

1h 44mins

12.48 sec

Not all issues were created for this run - 4 errors

Please note we ensured no other activity was done on the server while the import runs were executing.

Environment Details

Jira

v7.2.1 on Centos 7 (EC2) - using authentication on crowd

Confluence

5.10.2 on same server as jira - using authentication on crowd

JIRA CLI COnnector  

6.1.0

Run CLI Actions for JIRA

6.1.0-SNAPSHOT

Confluence CLI connector:  

6.1.0-SNAPSHOT

Run CLI Actions in Confluence

 6.1.0-SNAPSHOT

Database

Postgres 9.2.18

Some questions we need answered are

  • Is there a load limit on csv macro when processing more than a couple of import runs?

  • Is there any log kept by the plugins that we can make use of in identifying the cause of slowness?

  • Can we profile the import runs through csv macro somehow?

  • Any thoughts on the high CPU usage We noticed it doesn't go above 90%?

Environment

None

Activity

Show:
Ankit Dahiya
May 1, 2017, 5:01 AM

This was related to Groovy expressions in the workflow post-function and not related to CLI.

Asad Ahmed
April 14, 2017, 4:54 PM

Hi Sunita,

When importing data, sometime we get an exception. I have attached the trace from --verbose flag

Can you suggest why this occurs as the trace doesn't seem to help much. We know that the issue type is there and its valid for the project. Also it does not occur for other issues within the same file. Please note the files we are using have test data and we use them multiple times to import data, however this error occurs very rarely.

Ankit Dahiya
April 13, 2017, 1:00 PM

Hi Sunita,

Its true that there are multiple pages which this type of code.
Also think of it as same confluence page being accessed by multiple users or different confluence pages, each user can press the run button on their loaded confluence page (at same or different time, attaching same or different file, and while one request is being processed from one user, other request get queued.

Sunita Patro
April 13, 2017, 12:48 PM

Thanks for the storage format!
I am assuming, you have this code on different pages, and you are rendering them one after another almost parallel or with a second or two.

Ankit Dahiya
April 13, 2017, 12:05 AM

Hi Sunita, please see page below:

Invalid
Your pinned fields
Click on the next to a field label to start pinning.

Assignee

Bob Swift (Bob Swift Atlassian Apps)

Reporter

Asad Ahmed