bugGNU Octave - Bugs: bug #58252, [octave forge]...

 
 

bug #58252: [octave forge] (instrument-control) ustmc_read() does not read all the waveform data from oscilloscope

Submitter:  None
Submitted:  Sun 26 Apr 2020 11:44:27 PM UTC
   
 
Category:  Octave Package Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Incorrect Result
Status:  Need Info Assigned to:  None
Originator Name:  Walter Sjursen Originator Email:  -email is unavailable-
Open/Closed:  * Open Release:  * 4.2.2
Operating System:  * GNU/Linux Fixed Release:  None
Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Tue 05 May 2020 10:31:05 PM UTC, comment #17: 

If calling multiple reads gets back the whole thing then we can probally just loop the data read until theres nothing left


John Donoghue <lostbard>
Group Member
Fri 01 May 2020 02:38:20 PM UTC, comment #16: 


Looking at code
https://github.com/coreutils/coreutils/blob/master/src/head.c


Head potential calls read() multiple times

John Donoghue <lostbard>
Group Member
Fri 01 May 2020 02:22:40 PM UTC, comment #15: 

[1] In the code base for Linux, I found USBTMC_TIMEOUT is set to 5000ms (default). This is plenty of time for the scope to return its data.
[2] I ran a script from the Linux command line to talk directly with the scope as a character device "/dev/usbtmc0". The data was returned lightning fast (so fast, I did not realize at first that it did anything). Here is the script:

echo "*IDN?" > /dev/usbtmc0
head -n 1 /dev/usbtmc0 > scope.id
echo ":WAVEFORM:FORMAT ASCII" > /dev/usbtmc0
echo ":WAVEFORM:POINTS:MODE RAW" > /dev/usbtmc0
echo ":WAVEFORM:PREAMBLE?" > /dev/usbtmc0
head -n 1 /dev/usbtmc0 > scope.preamble
echo ":WAVEFORM:DATA?" > /dev/usbtmc0
head -n 1 /dev/usbtmc0 > scope.data

It only took one read (last head command) to retrieve all the waveform data from the scope. The script sends all its output to files:

-rw-rw-r-- 1 waltersjursen waltersjursen 875010 Apr 30 19:35 scope.data
-rw-rw-r-- 1 waltersjursen waltersjursen     62 Apr 30 19:35 scope.id
-rw-rw-r-- 1 waltersjursen waltersjursen      5 Apr 30 20:52 scope.points
-rw-rw-r-- 1 waltersjursen waltersjursen     85 Apr 30 19:35 scope.preamble



Walter Sjursen <wsjursen>
Fri 01 May 2020 02:17:45 PM UTC, comment #14: 

Oh - it looks like it found the tmc.h file (/usr/include/linux/usb/tmc.h on my machine) but the timeout defines it was trying to use were not there.


John Donoghue <lostbard>
Group Member
Fri 01 May 2020 02:07:15 PM UTC, comment #13: 

[1] usbtmc function print out CAPS: F7
[2] I tried to set a timeout but received an error message (see below). But the scope returns data very fast (I will post a new comment on that).
[3] I tried multiple reads. It took 15 reads to get all the scope's data (see output of whos command below). Variables data_raw* is the data if each sequential read. Note there is not a consistent size. Variable data_raw stiches together all the data. that is data_raw = [data_raw1 data_raw2 ... data_raw15]. The stitched-together data looks correct.


>> scope = usbtmc(', bu/dev/usbtmc0')

CAPS: F7
scope = 36

>> usbtmc_timeout

error: Invalid call to usbtmc_timeout.  Correct usage is:

 -- Loadable Function: TIMEOUT = usbtmc_timeout(USBTMC)
 -- Loadable Function: RETCODE = usbtmc_timeout(USBTMC, TIMEOUT)

Additional help for built-in functions and operators is
available in the online version of the manual.  Use the command
'doc <topic>' to search the manual index.

Help and information about Octave is also available on the WWW
at http://www.octave.org and via the -email is unavailable-
mailing list.

>> timeout = usbtmc_timeout(scope)

warning: timeout not implemented
timeout = 0

>> retcode = usbtmc_timeout(scope,10000)

warning: timeout not implemented
retcode = -1

>> retcode = usbtmc_timeout(scope,10)

warning: timeout not implemented
retcode = -1


>> whos

Variables in the current scope:

   Attr Name             Size                     Bytes  Class
   ==== ====             ====                     =====  =====
        Xtime            1x7680                   61440  double
        Xtrigger         1x1                          8  double
        Ydata            1x7680                   61440  double
        ans              1x1                          8  double
        chan             1x1                          8  double
        cmd              1x15                        15  char
        count            1x1                          8  double
        dataFormat       1x10                        80  double
        data_raw         1x107530                107530  uint8
        data_raw1        1x3804                    3804  uint8
        data_raw10       1x8204                    8204  uint8
        data_raw11       1x8204                    8204  uint8
        data_raw12       1x8204                    8204  uint8
        data_raw13       1x8204                    8204  uint8
        data_raw14       1x8204                    8204  uint8
        data_raw15       1x8204                    8204  uint8
        data_raw16       1x5096                    5096  uint8
        data_raw2        1x2                          2  uint8
        data_raw3        1x8286                    8286  uint8
        data_raw4        1x2                          2  uint8
        data_raw5        1x8300                    8300  uint8
        data_raw6        1x8204                    8204  uint8
        data_raw7        1x8204                    8204  uint8
        data_raw8        1x8204                    8204  uint8
        data_raw9        1x8204                    8204  uint8
        data_str         1x107520                107520  char

Walter Sjursen <wsjursen>
Thu 30 Apr 2020 10:38:12 PM UTC, comment #12: 

the test version should also print out the api version and capabilities when it creates the device.

It only do that if it could compile with usb/tmc.h for the ioctls, otherwise even setting the timeouts wont work.

John Donoghue <lostbard>
Group Member
Thu 30 Apr 2020 10:32:37 PM UTC, comment #11: 

altenatively, if you try a second read after the first one (when there is more points that currently read) does it read more data ?

John Donoghue <lostbard>
Group Member
Thu 30 Apr 2020 10:29:00 PM UTC, comment #10: 

Did you try getting the current timeout value and then setting a bigger timeout value in octave ?

John Donoghue <lostbard>
Group Member
Thu 30 Apr 2020 09:01:12 PM UTC, comment #9: 

One more experiment...
from the linux command line:

$echo ":WAV:DATA?" > /dev/usbtmc0
$head -c 107530 /dev/usbtmc0 > scope.data
ls -l scope.data
-rw-rw-r-- 1 ......  107530 Apr 30 16:28 scope.data
$head -c 37 scope.data
#800107519-6.28141e-002,-8.79397e-002

107530 characters were read into scope.data
the first 10 characters show: "#800107519" indicating there are 107519 waveform data points following. The above head command showed the header and the first two data points (14 characters per data point including the "," separator).

10 characters for header,
107519 characters for data (7680 data pts x 14 char/pt)
-1 character (the last data point is not followed by a delimiter
+1 "\n" character at the end of the data
+1 end-of-file character
==============================
1077530 bytes (all the scope data was read in)

figure3.png is a plot of the 10Vpk, 3kHz sine-wave:

This demonstrates the scope, computer hardware, linux and usbtmc.ko all seem to be working okay.






Walter Sjursen <wsjursen>
Thu 30 Apr 2020 08:05:53 PM UTC, comment #8: 

I have now run the test version of instrument-control. Same results. Below 280 waveform points, it works fine. 280 and above, it does not return all the data. See attached figure2.png


Walter Sjursen <wsjursen>
Thu 30 Apr 2020 04:58:39 PM UTC, comment #7: 

I did not try the test version of instrument-control yet. I did run a test where I increased the number of waveform points requested and plot the number of returned points vs. request points (see the attached file "figure1.png"). At about 280 waveform points (3930 bytes of return data + header), it starts falling apart. Below 280 waveform points, it work's fine.


Walter Sjursen <wsjursen>
Thu 30 Apr 2020 01:19:51 AM UTC, comment #6: 

Ive added a usbtcm_timeout function to this test version of instrument control - i dont know it will ame any difference but may help

(file #48970)

John Donoghue <lostbard>
Group Member
Wed 29 Apr 2020 02:11:15 AM UTC, comment #5: 

I might be able to write a test program in c++ that you can test a couple of the settings if that will help, it can dump out some more info.

John Donoghue <lostbard>
Group Member
Wed 29 Apr 2020 01:58:35 AM UTC, comment #4: 


The  instrument control toolkit just uses the default values of the device, and does no processing or anything, just calls read and write to the device fd.

So the entry the log further indicates to me that its not a instrument control issue (as such), but rather driver or new firmware related.

Since instrument control is using default, perhaps it is timing out before it gets all the data, or the driver doesn't support the amount data being sent.

I don't use the usmtcm driver myself, so maybe someone else has used it and can answer better.

I see on keysite that there is an old driver with some additional tools for setting timeouts etc that may be useful (assuming they still compile)

https://www.keysight.com/upload/cmc_upload/All/usbtmc.htm?&cc=US&lc=eng

Or possibly could change the device open instrument code to set a different timeout value if supported.

If you set lower resolution is there a setting that does work, that would at least prove everthing else is working?

John Donoghue <lostbard>
Group Member
Wed 29 Apr 2020 01:10:38 AM UTC, comment #3: 

Some additional info...
When the problem occurs, I see the following dmesg "reply with wrong bTag":

[ 1466.591780] usb 1-1: new high-speed USB device number 4 using xhci_hcd
[ 1466.623067] usb 1-1: New USB device found, idVendor=2a8d, idProduct=1797, bcdDevice= 1.00
[ 1466.623074] usb 1-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 1466.623077] usb 1-1: Product: DSO-X 1102G
[ 1466.623080] usb 1-1: Manufacturer: Keysight Technologies
[ 1466.623083] usb 1-1: SerialNumber: CN56526430
[ 1466.642947] usbcore: registered new interface driver usbtmc
[ 2128.023156] usbtmc 1-1:1.0: Device sent reply with wrong bTag: 9 != 14

Anonymous
Tue 28 Apr 2020 10:59:52 PM UTC, comment #2: 

%% Octave version:

>> version

ans = 4.2.2

%% Octave packages installed:

>> pkg list

Package Name        | Version | Installation directory
--------------------+---------+-----------------------
           control *|   3.2.0 | /home/waltersjursen/octave/control-3.2.0
instrument-control *|   0.5.0 | /home/waltersjursen/octave/instrument-control-0.5.0
            signal  |   1.4.1 | /home/waltersjursen/octave/signal-1.4.1

%% Get an FID for the oscilloscope:

>> scope=usbtmc('/dev/usbtmc0')

scope = 40

%% run a script to get the waveform from the oscilloscope:

>> test3

nChar = 107530 % this is the number of characters that are requested in the usbtmc_write().

>> whos

Variables in the current scope:

   Attr Name             Size                     Bytes  Class
   ==== ====             ====                     =====  =====
        Xtime            1x7680                   61440  double
        Xtrigger         1x1                          8  double
        Ydata            0x0                          0  double
        ans              1x1                          8  double
        chan             1x1                          8  double
        cmd              1x15                        15  char
        count            1x1                          8  double
        dataFormat       1x10                        80  double
        data_raw         1x2                          2  uint8
        data_str         1x0                          0  char
        format           1x1                          8  double
        mode             1x4                          4  char
        nChar            1x1                          4  uint32
        nFormatRead      1x1                          8  double
        nRead            1x1                          8  double
        points           1x1                          8  double
        scope            0x0                          0  octave_usbtmc
        type             1x1                          8  double
        xincrement       1x1                          8  double
        xorigin          1x1                          8  double
        xreference       1x1                          8  double
        yincrement       1x1                          8  double
        yorigin          1x1                          8  double
        yreference       1x1                          8  double

Total is 7727 elements using 61665 bytes

%% Q. What is the output from usbtmc_write(scope,':WAV:PRE?') read?
%% A. Here is a snippet of test3.m

usbtmc_write(scope,':WAV:PRE?');
[dataFormat, nFormatRead] = usbtmc_read(scope, 200);
dataFormat = str2num(char(dataFormat));

format     = dataFormat(1);
type       = dataFormat(2);
points     = dataFormat(3);
count      = dataFormat(4);
xincrement = dataFormat(5);
xorigin    = dataFormat(6);
xreference = dataFormat(7);
yincrement = dataFormat(8);
yorigin    = dataFormat(9);
yreference = dataFormat(10);

%% and the data it returned:

>> nFormatRead

nFormatRead =  86       % number of characters read (returned)

>> dataFormat

dataFormat =

   4.0000e+00   3.0000e+00   7.6800e+03   1.0000e+00   2.6042e-07  -1.0000e-03   0.0000e+00   7.8518e-04   0.0000e+00   3.2768e+04

%% interpreting the returned data:
format     = 4           % return data format (WORD|BYTE|ASCII) The manual does not define "4", but the data was returned in ASCII.
type       = 3           % the acquisition mode of the scope is HiRes
points     = 7680        % number of data points available from the scope
count      = 1           % 1 indicates not in Average mode
xincrement = 2.6042e-07  % sample rate = 1/xincrement
xorigin    = -0.0010000  % first data point is 1ms before the trigger point
xreference = 0           % index of data at xorigin
yincrement = 7.8518e-04  % resolution of y-axis
yorigin    = 0           % y-axis value at center of oscilloscope screen
yreference = 32768       % data point value at yorigin, but not meaningful if data is in ASCII format
 

%% Q. What is the output from the ":WAVeform?" query?
%% A. There is no generic ":WAVeform?" query defined.


%% Q. What is the output from the ":ACQuire:POINts?" query?
%% A. After running the 'test3.m' script, I manually queried the scope. It returned:

>> acq_points = usbtmc_write(scope,':ACQuire:POINTs?')

acq_points =  16



%% Q. What is the output data from usbtmc_write(scope,':WAV:DATA?') read when it returns only 2 bytes?
%% A. The code snippet to read the data and the return data follow:

usbtmc_write(scope,':WAV:DATA?');
nChar = uint32(10 + 14*points);
nChar=uint32(nChar);
[data_raw, nRead] = usbtmc_read(scope,nChar);
data_str = char(data_raw(11:length(data_raw)));

%% The returned data is:

>> nRead

nRead =  2

>> data_raw

data_raw =
  35  5

>> char(data_raw)

ans = #8

>>


%% Note that the "#8" is the beginning of the header that defines how many characters are going to be returned.
%% Return data has the form: #800107519-2.26131e-001,-2.26131e-001,-2.01005e-001,-2.26131e-001,-1.75879e-001,...
%% where the '#' is the start of the header, the next character '8' indicates the next 8 characters provide the number of bytes (characters)
%% of data that follow. In this example, the first data point is -0.226131 volts.
 

%% I rebooted the system, added a few extra statements into the test3.m script and ran it again. This time it returned more than 2 characters,
%% but still not all.

%% new lines of code before reading the waveform data:

% added for debugging...
acq_points = usbtmc_write(scope,':ACQuire:POINTs?')
wav_points_mode = usbtmc_write(scope,':WAVeform:POINTs:MODE?')
wav_points = usbtmc_write(scope,':WAVeform:POINTs?')

acq_points =  16
wav_points_mode =  22
wav_points =  17

>> nChar

nChar = 107530

>> whos

Variables in the current scope:

   Attr Name                 Size                     Bytes  Class
   ==== ====                 ====                     =====  =====
        Xtime                1x7680                   61440  double
        Xtrigger             1x1                          8  double
        Ydata                1x571                     4568  double
        acq_points           1x1                          8  double
        ans                  1x1                          8  double
        chan                 1x1                          8  double
        cmd                  1x15                        15  char
        count                1x1                          8  double
        dataFormat           1x10                        80  double
        data_raw             1x8004                    8004  uint8
        data_str             1x7994                    7994  char
        format               1x1                          8  double
        mode                 1x4                          4  char


%% It now read 8004 of the 107530 characters which is 571 of 7680 data points. :-(

Anonymous
Mon 27 Apr 2020 11:24:06 AM UTC, comment #1: 

The code hasnt changed for usbtmc for a long time, so it probally isnt changes for instrument control that causing your issues.

The class effectively just passes to an reads from the device with the only interaction being that it converts to / fom octave types.

So what is the output from

usbtmc_write(scope,':WAV:PRE?') read?
And few others:

:WAVeform?

:ACQuire:POINts?

:WAVeform:FORMat?

Whats the output data from

usbtmc_write(scope,':WAV:DATA?') read when it returns only 2 bytes?


From the response header for WAV:DATA "#800107519", that would be 1 million bytes for the wave data - what size where you trying to read in ?






John Donoghue <lostbard>
Group Member
Sun 26 Apr 2020 11:44:27 PM UTC, original submission:  

This appears to be an instrument-control problem.

Computer and OS
$uname -a
Linux Dell-XPS-15-9560 5.3.0-46-generic #38~18.04.1-Ubuntu SMP Tue Mar 31 04:17:56 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Octave version 4.2.2
Packages:
Package Name        | Version | Installation directory
--------------------+---------+-----------------------
           control *|   3.2.0 | /home/.../octave/control-3.2.0
instrument-control *|   0.5.0 | /home/.../octave/instrument-control-0.5.

I am connecting to a Keysight model DSO-X 1102G oscilloscope through a USB connection as a usbtmc class device (/dev/usbtmc0).
When trying to read waveform data, only part of the waveform data is read. I am using:
[data, nRead] = usbtmc_write(device,nChar)
It appears if nChar is larger than the "10 + number of samples" the function only returns 2 bytes. Since I am trying to return ASCII data, it needs 14 bytes per sample of data;

>> char(data_raw(1:10+5*14-1))

ans = #800107519-2.26131e-001,-2.26131e-001,-2.01005e-001,-2.26131e-001,-1.75879e-001

see attached test2.m octave script for further details.

With an older version of instrument-control it used to return all the data.
Since then I upgraded the computer from Unbuntu 16.x to 18.04
The firmware in the scope was upgraded
And of course instrument-control was updated.

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #48979:  figure3.png added by wsjursen (42KiB - image/png)
file #48978:  figure2.png added by wsjursen (42KiB - image/png)
file #48976:  figure1.png added by wsjursen (46KiB - image/png)
file #48970:  instrument-control-0.5.0.tar.gz added by lostbard (483KiB - application/x-gzip)
file #48948:  return_data.txt added by None (105KiB - text/plain - using echo ":WAV:DAT?">/dev/usbtmc0 and cat /dev/usbtmc0>return_data.txt I receive ALL the correct ASCII data.)
file #48947:  test2.m added by None (2KiB - text/x-objcsrc)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by wsjursen (Updated the item)
  • -email is unavailable- added by lostbard (Posted a comment)
  • -email is unavailable- added by siko1056 (Updated the item)
  • -email is unavailable- added by None (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 8 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-04-30 wsjursen Attached File- Added figure3.png, #48979
    2020-04-30 wsjursen Attached File- Added figure2.png, #48978
    2020-04-30 wsjursen Attached File- Added figure1.png, #48976
    2020-04-30 lostbard Attached File- Added instrument-control-0.5.0.tar.gz, #48970
    2020-04-28 lostbard StatusNone Need Info
    2020-04-27 siko1056 SummaryInstrument-control ustmc_read() does not read all the waveform data from oscilloscope [octave forge] (instrument-control) ustmc_read() does not read all the waveform data from oscilloscope
    2020-04-27 None Attached File- Added return_data.txt, #48948
    2020-04-26 None Attached File- Added test2.m, #48947

    Back to the top

    Powered by Savane 3.13-758e.
    Corresponding source code