Search


Ploticus > Scripts >

proc getdata

proc getdata gets data for plotting, and must be invoked before plotting any data. Data may be read from files, from commands, or specified directly, and various input data formats are supported. The data which are read become the current data set that subsequent plotting procs will access, and replace any data that were read in (or generated) previously.

proc getdata will set the variable NRECORDS to the number of data rows gotten. The variable NFIELDS will be set to the number of fields per data row.

Use showdata: yes to check that ploticus is parsing your data as expected.

See also

Discussion of input data formats and parsing rules

Discussion of working with multiple data sets

Attributes

Be sure and set the delim attribute appropriately when using field delimitations other than whitespace-delimited.

Specifying a data source

One (and only one) of the following data sources must be specified:

pathname     filename

Name of a file containing plotting data. The file will be opened directly.

Security concern: user-supplied values, such as CGI user variables, should not be used to build the pathname, unless proper measures are taken to detect and remove the ../ construct (used as a hack to see higher levels of the file system).

file     filename

Shell-expandable name of a file containing plotting data. This name will be used along with cat(1) in a shell command, thus exported shell variables and metacharacters may be part of the name. A dash (-) may be used if data is to be read from the standard input, (or the standardinput attribute may be used).

Security concern: user-supplied values, such as CGI user variables, should not be used to build filename.

On win32 platforms, or when operating in direct CGI mode (2.31-08+) filename automatically maps to pathname .

Example: filename: myfile.dat

data     multiline-text

Literal specification of plotting data. Terminates at first blank (empty) line. Best practice is for rows to not have any leading whitespace, and for the first row of data to be on the line following the data tag, as shown below. The first row of data may be a field name header. Use showdata: yes to verify that your data are being input and parsed as expected. Example:

data:   
Case_1 0 4 4.1   
Case_2 1 5 4.4  
Case_3 2 2 4.0 
Case_4 3 9 4.8

command     shell command line

An external shell command that will produce plot data on its standard output.

Security concern: user-supplied values, such as CGI user variables, should not be used to build the shell command. If this must be done, use #shell / #endshell along with data:, which provides some protection against shell metacharacter hacks.

Example: command: cat mydat | uniq -c

commandmr     multi-row shell command

Same as command but shell command can occupy multiple rows, and is is terminated by a blank line. Command can be up to ~1000 characters long.

Security concern: user-supplied values, such as CGI user variables, should not be used to build the shell command. If this must be done, use #shell / #endshell along with data:, which provides some protection against shell metacharacter hacks.

You can overlay flow-of-control directives as in this example:

  commandmr:  cat foo.dat |
              process_step1 |
      #if @sortmode = 1
              sort +2 -3
      #else
              cat -
      #endif
 

standardinput     yes | no

If yes, data stream is read from the ploticus standard input.

sql     sql command

Submit the given sql command and capture the results as tabular data, using sql result column names as the field names. This is currently available only for builds related to the QUISP/SHSQL package.

sqlmr     multi-row sql command

Same as sql but SQL command can occupy multiple rows, and is is terminated by a blank line. Command can be up to ~1000 characters long. You can overlay flow-of-control directives as in this example:

  sqlmr: select id, lastname, firstname, balance
         from accounts
    #if @mode = 1
         where acct_type = g
    #endif
      

#intrailer

Indicates that a data attribute will be given in a #proc trailer, at the end of the script file. See EXAMPLES, below.

Characteristics of the data stream

delim     whitespace | spacequote | comma | tab | bar | auto

Field delimitation used in the data set. Default is spacequoted. Use auto to have ploticus attempt to automatically determine the delimitation (2.40+). See dataformat for details.
Example: delim: comma

nfields     n

If specified, this sets the expected number of fields per record. If a data row has more than the expected number of fields, extra fields are silently ignored. If a data row has less than the expected number of fields, blank fields are silently added. This is applied after any filter processing. If not specified, the first non-comment non-header row encountered will set the expected number of fields.

commentchar     string

A character or group of characters that is used to signify a comment in the data file. Commented lines will be skipped. Default is //. Example: commentchar: #

Setting field names

fieldnameheader     yes | no

Allows field names to be embedded in the input data. If yes, the first non-comment line in the data is expected to contain field names. This line is not considered part of the data. The field name header should be delimited like the rest of the data. Field names may not contain embedded white space, commas, or quote characters but (2.30+) there is a way to encode spaces and commas... see proc settings encodenames.

fieldnames     namelist

Specify field names for input data fields. namelist is a list delimited by spaces and/or commas. Names may include any alphanumeric characters with a maximum length of 48, and are case-insensitive. Field names may not contain embedded spaces or commas, but (2.30+) you can encode them... see proc settings encodenames. Note: if you are using a filter (see below) you may want to use pf_fieldnames (see below) to name the result fields.
Example: fieldnames: date group n

fieldnamerows     multiline text

Same as fieldnames (see above), except that field names are given one per line. Must be terminated by a blank line. Example:


fieldnamerows:
  id
  type
  age
  sex

pf_fieldnames     namelist

Assign new field names to filter result. See filter attribute described below. Useful when the filter result has a different logical record format than the input. See also fieldnames above.
Example: pf_fieldnames: date z sum1 sum2

Development & debugging

showdata     yes | no
(old name showresults)

If yes, the data as read, parsed, and filtered are written to the diagnostic file in bar-delimited format.

Selecting and manipulating input rows

select     select expression

This allows certain data records to be selected for inclusion based upon a selection expression. Incoming data fields are referenced by number, using a double at-sign (@@) prefix. Hint: use the showresults attribute when debugging.
Example: select: @@3 = g
This would select all data records having 3rd field equal to g.

samplerate     n

Keep every Nth data row and throw away everything else. This can be used when working with very large data sets, since often a plot with 10,000 data points looks the same as one with 10 million data points. New in 2.42

filter     multiline text

An embedded script which allows flexible processing to be applied to incoming data records one at a time. Typical uses are for concatenating or splitting fields, doing on-the-fly date conversions, or generating derived fields such as the sum of several fields or the difference between two fields.

The embedded script will be applied once to every incoming data record. The script should produce some "output"; generally the last statement is a ##print. The output must use the same delimitation method as the input. The output may have a different logical record format than the input. If you are using field names, the pf_fieldnames attribute (see above) may be used to name the filter result fields when result record format differs from that of the input.

The script uses the same syntax as the greater ploticus script, except that:
directives must begin with two pound signs (##) instead of one
local variables begin with two at signs (@@) instead of one
fields on the incoming data record are accessed like this: @@1 for the first field, @@2 for the second, etc. If you are using field names, these may be used as well, eg: @@score.
the only directives that may be used are ##set, ##if, ##elseif, ##else, ##print, ##call, and ##exit

Other things worth noting:
the filter script is terminated using a blank line.
use the showresults attribute when debugging.
if filter is used along with select, the select is applied first.
filter cannot be used on data specified using the data attribute.
filter can only access fields from one data record at a time.
variables from the ploticus script may be referenced within the filter script (use one at-sign @). Evaluation occurs before the filter script executes.
ploticus script #if/#else statements (single pound sign) may be used to selectively execute portions of the filter script. Interpretation occurs before the filter script executes.
Example: This filters out data records having field 2 or field 3 equal to M. It then calulates the difference in days between two dates and puts this difference in the variable DIFF. Finally it "prints" incoming field 1 along with DIFF. Thus the result of this #proc getdata will have be data records having two fields.

filter:
     ##if @@2 = M || @@3 = M
       ##exit
     ##endif
     ##set DIFF = $daysdiff(@@3,@@2)
     ##print @@1 @@DIFF

There are several more filter examples in the FAQ







More examples

Data specification may be located at the end of the script file by using #intrailer and #proc trailer. This may be useful in "getting the data out of the way", or with automated building of script files. Here is how this is done:

#proc getdata
#intrailer

other #procs, etc.

#proc trailer
Data:	0.3 0.5 2.3
	3.5 9.4 1.4
	..etc..
end of file





Variables that are set by proc getdata

NRECORDS = the number of data rows gotten

NFIELDS = the number of fields per record






Hints

The processed, filtered data set result can be written to a tab-delimited file by doing something like this:

 #proc getdata
 ...
 #endproc
 #proc processdata 
 action: echo
 outfile: /tmp/myfile





It is possible to set working variables from within the data file.



During debugging, set showresults: yes in order to see the data after it is read and parsed. Especially useful when working with filter.



In dynamic content environments it's good to gracefully handle the situation of an empty data file or command that produced no output. Example:

#proc getdata
   ...

#proc endproc
#if @NRECORDS = 0
  #proc annotate
  location: 3 3
  text: No data found.

  #exit
#endif