Cancel
Showing results for 
Search instead for 
Did you mean: 

Re: MCL V3 How Can I speed up the data ingestion ?

Siemens Creator Siemens Creator
Siemens Creator

I set the debug level to INFO 

Uploading for 7 Datapoints takes 1.07 s.

In the output of my agent it is exactly the time from libcurl send to evalute response. in the debug trace.

For 262 datapoints it is 32.52s

It seems obvious that Mindsphere is consuming the time.

Or is there some time consuming sub on MCL side between send http request and evalute response ?

 

see attachment for details.

Re: MCL V3 How Can I speed up the data ingestion ?

Experimenter
Experimenter

I think you have managed to exchange requsest in following format -

For Example : 

[
    {
        "timestamp": "2017-02-01T08:30:03.780Z",
        "values": [
            {
                "dataPointId": "{{_datapoint_id_1}}",
                "value": "9856",
                "qualityCode": "0"
            },
            {
                "dataPointId": "{{_datapoint_id_2}}",
                "value": "3766",
                "qualityCode": "0"
            }
        ]
    },...
]

 In my case i tried to exchange a timeseries data with 50 value sets having 4 datapoint which resulted to MCL_STORE_ITEM_EXCEEDS_MAX_HTTP_REQUEST_SIZE error. Is there any way to check the size before we exchange the store?

Issue posted on following thread.
https://community.plm.automation.siemens.com/t5/Developer-Space/Is-there-any-way-to-check-store-size...

Re: MCL V3 How Can I speed up the data ingestion ?

Valued Contributor
Valued Contributor

Hello @HorstRieger ,

Debug level logs would reveal the root cause that causes delay.

Can you please share it?

Re: MCL V3 How Can I speed up the data ingestion ?

Siemens Creator Siemens Creator
Siemens Creator

Debug Level = debug (2)

80 Datapoints

10.468s duration

see attached file : Debug.txt

Re: MCL V3 How Can I speed up the data ingestion ?

Valued Contributor
Valued Contributor

The lines below show where the delay occurs:

 

29108: 0119-06-13 16:31:47,652 | 02070 | 02070 | DEBUG | [..] | http_client_libcurl._curl_debug_callback | 553 | => Send data, 0000015241 bytes (0x00003B89)
...
29586: 0119-06-13 16:31:47,652 | 02070 | 02070 | DEBUG | [..] | http_client_libcurl._curl_debug_callback | 553 | 696861436E33642D2D
29587: 0119-06-13 16:31:57,536 | 02070 | 02070 | DEBUG | [..] | http_client_libcurl._curl_debug_callback | 553 | <= Recv SSL data, 0000000005 bytes (0x00000005)

Time period between the beginning of Send and the beginning of Receive is 9,884 seconds.

I think this is caused by slow upload speed that may be caused by slow network speeds or low system resources.

 

I made a timeseries upload test with cable connection (~90Mbps upload speed) and 2G mobile connection (~0.1Mbps upload speed) and noted down the upload time periods for 1000 datapoints:

cable connection: 0,371 seconds

2G mobile connection: 23,316 seconds

 

I think it is better to try cable connection rather than wifi with Raspberry Pi.

As an alternative you may try sharing your mobile phone's LTE connection with Pi via Wi-Fi.

Can you please repeat the test with cable (LAN) or LTE mobile connections?

Re: MCL V3 How Can I speed up the data ingestion ?

Siemens Creator Siemens Creator
Siemens Creator
I'll try this with a wired connection.
I'll let you know about the results.
Thx !

Re: MCL V3 How Can I speed up the data ingestion ?

Siemens Creator Siemens Creator
Siemens Creator

I could make a test with a wired connection and now it is 262 datapoints in 9,23 s with the Rpi3B+

Obviously the wifi connection is (unexpectedly) slow...