Search
Ploticus >
Scripts >
proc processdata performs various types of manipulations on the
current data set.
Here are some of the things this proc can do:
- accumulation (horizontal and vertical)
- count/summate on instances, numeric bins, or time-based bins
- rotation of row/column matrix
- find mean, sd, min, max
The action attribute controls the type of processing to be done.
The result is usually a new in-memory data set that subsequent plotting procs work from,
but sometimes the result is simply the setting of some variables, as with action: totals,
and it is also possible to write the result to a file (see outfile).
When a new data set is created, it will be kept in memory along with the original data by default.
To discard the new data set and return to the original one, use
proc usedata.
Some of the actions assign default field names to the result data set as noted, others do not
(result names can always be overridden by specifying fieldnames).
Use showresults to verify that any field names associated with result data set are as expected.
For additional data processing capabilities see also
proc getdata
filter
and
proc tabulate.
Example
A gallery example that demonstrates and tests all the available actions is
processdata
Variables that are set by processdata
NRECORDS = Number of rows in the data result.
NFIELDS = Number of fields per row in the data result.
TOTALS = If totals, percents, or accumulation are being done, this variable is set to the column (field) total(s). See
action: totals for more info.
action: stats and action: breaks set additional variables, described in the stats section below.
Actions
The action attribute controls the type of operation that will be done.
Often the fields attribute is used to indicate which field(s) are involved.
If action is not specified, it defaults to echo.
action: accumulate
Rewrite field(s) as cumulative series (downward accumulation); all other fields are passed through transparently.
The field(s) to operate on must be given in the fields attribute.
The TOTALS variable is also set to the column (field) total, in case it's useful.
For example, the data set on the left would be transformed to the one
on the right (fields: 2):
A21 3 A21 3
A22 5 --> A22 8
A23 2 A23 10
A24 1 A24 11
action: breaks
performs break processing, which is
useful for stepping through a sorted data set one chunk at a time and producing a plot for each chunk.
Input data must be ordered on (or at least clustered on) the break fields.
Specify fields as one or more (max 5) identifying "break" fields.
The complen attribute (see below) can be used to control the number of characters examined in comparisons.
Processing involves examining data rows one at a time, and when a difference in any of the break fields is detected
processing stops. Generally used within a loop.
See further discussion below
action: breakreset
Reset the "current row" to the beginning of the data set,
for the occasional time when more than one pass through a data set will be done
using action: breaks .
This is automatically done if a new data set is read by proc getdata.
action: count
Count instances or perform summations by instance.
An "instance" can be a literal string, a substring (specify complen),
or a numeric or chronological range (specify binsize and perhaps binmod).
Input data must be sorted (or at least grouped) on the key field.
Resulting data set will always have two fields.
One or two fields must be specified using the fields attribute.
If one field is specified, the result fields will be named bin and count,
for example (fields: 1):
062698 062698 2
062698 --> 062898 1
062898 070198 3
070198 070498 1
070198
070198
070498
If two fields are specified, the result fields will be named bin and sum,
for example (fields: 1 2):
062698 4 062698 10
062698 6 --> 062898 3
062898 3 070198 9
070198 2 070498 2
070198 4
070198 3
070498 2
See also action: segment and action: segmentb described below, which are
similar.
A gallery example that uses action: count is
hitcount
action: echo
Pass data rows transparently to the new data set, except for any effect of select, keepfields, or rejectfields.
action: join
Perform a relational join operation. (2.30+)
All records come from the current data set; left side records are selected using the leftselect
attribute and right side records are selected using rightselect.
The records are joined on one or more fields specified in the fields attribute.
Input data must be sorted on these fields (integer fields should be ordered numerically; alphanumeric
fields should be ordered alphabetically).
One (and only one) left side record and one (and only one) right side record must exist in order to be joined
(but see leftjoin and rightjoin below).
For example, suppose we want a scatterplot where the X component and Y component
are represented in different sets of records within the data set:
001 X 4.3 001 X 4.3 001 Y 5.2
001 Y 5.2 --> 002 X 3.2 002 Y 2.9
002 X 3.2
002 Y 2.9
In order to do a scatterplot of X vs. Y we need to do a join so that all data for case 001 are
on the same row, and all data for case 002 are on the same row. We can do this using:
#proc processdata
showdata: yes
action: join
fields: 1
leftselect: @@2 = X
rightselect: @@2 = Y
Also available are action: leftjoin and action: rightjoin.
With a left join, if a left side record exists with no right side companion, the right side
is filled in with a missing data code (= by default; can be specified in missingdatacode
attribute). A right join is the same thing but in reverse.
action: numberrows
Prepend a row number field to each row. For example:
A AA AAA 0 1 A AA AAA 0
B BB BBB 0 ---> 2 B BB BBB 0
C CC CCC 1 3 C CC CCC 1
D DD DDD 1 4 D DD DDD 1
E EE EEE 0 5 E EE EEE 0
F FF FFF 1 6 F FF FFF 1
action: percents
Rewrite one or more fields as percentages of its field (column) total;
all other fields are passed through transparently.
The field(s) to operate on must be given in the fields attribute.
The TOTALS variable is also set to the column (field) total.
For example (fields: 1):
8 40
4 --> 20
3 15
5 25
action: reverse
The last record becomes the first one; the
record order is reversed. For example:
AXB 34 DIF 14
BYA 22 --> CES 52
CES 52 BYA 22
DIF 14 AXB 34
action: rotate
First row becomes 1st field, 2nd row
becomes 2nd field, and so on. This may be useful
in that most of the plotting procs work from data fields,
but sometimes data is given (or is more intuitive) in rows.
For example:
A 2 4 6 8 10 --> A B
B 3 6 9 12 15 2 3
4 6
6 9
8 12
10 15
action: segment
Similar to action: count, except that instead of a count result, a range is produced,
useful for plotting bar segments, etc.
Resulting data set will always have three fields: 1) key field, 2) start position, 3) end position.
If one field is specified, the start and end position will be record numbers of the first
and last records in the range (where first record is 1). The result field names will be bin startrow endrow.
For example (fields: 1):
062698 062698 1 3
062698 ---> 070198 4 6
062698 070498 7 7
070198
070198
070198
070498
If two fields are specified, the start and end position will be the contents of the second
specified field, for the first and last records in the range. The result field names will be bin startval endval.
For example (fields: 1 2):
062698 A 062698 A C
062698 B ---> 070198 D F
062698 C 070498 G G
070198 D
070198 E
070198 F
070498 G
action: segmentb
Same as action: segment, except that groups will butt up against one another.
For example (fields: 1 2):
062698 A 062698 A D
062698 B ---> 070198 D G
062698 C 070498 G G
070198 D
070198 E
070198 F
070498 G
action: summary
Produce a new data set containing summary statistics on the input, clustering on specified "break" fields.
Specify valfield as the data field containing numeric values to be examined.
Specify fields as one or more identifying "break" fields.
The complen attribute (see below) can be used to control the number of characters examined in comparisons.
Processing involves examining data rows one at a time, and when a difference in any of the break fields is detected
the mean and other stats are computed on valfield for the current group of records, and one result row is added to the new data set.
The result data set will always have the following fields, named as shown:
id1, (id2 etc.), mean, sd, sem, n_obs, min, max, sum.
Input data must be sorted (or at least clustered) on the break fields.
(New in version 2.40)
Here's an example (fields: 1 2 3 and valfield: 5):
2909 WSB/EiJ f 1 1
2909 WSB/EiJ f 2 3 ----> 2909 WSB/EiJ f 2.25 0.886405 0.313392 8 1 3 18
2909 WSB/EiJ f 3 3 2909 WSB/EiJ m 2.6 0.894427 0.4 5 1 3 13
2909 WSB/EiJ f 4 3
2909 WSB/EiJ f 6 2
2909 WSB/EiJ f 7 2
2909 WSB/EiJ f 8 3
2909 WSB/EiJ f 9 1
2909 WSB/EiJ m 13 3
2909 WSB/EiJ m 15 3
2909 WSB/EiJ m 16 3
2909 WSB/EiJ m 18 3
2909 WSB/EiJ m 19 1
In the above example, the result data set would then have fields named:
id1 id2 id3 mean sd sem n_obs min max sum
action: summaryplus
Same as action: summary above, except that median, and 5th, 25th, 75th, and 95th percentiles are computed as well.
(New in version 2.40)
The result data set will always have the following fields, named as shown:
id1, (id2 etc.), mean sd se n_obs min max sum pctl5th pctl25th median pctl75th pctl95th
.ig >>
action: stats
Scan all eligible data rows and
find the mean, SD, min, max, etc. for specified data field(s), and put mean, SD, min, max (etc) into ploticus variables.
No new data set is created.
The data field(s) to operate on should specified in the fields attribute.
Sets the following ploticus variables:
MEAN = mean of all numeric values found
SD = standard deviation
N = number of numeric values found
MIN = the lowest numeric value found
MIN_ID = contents of identifier/abscissa field in data row where minima was found
MAX = the highest numeric value found
MAX_ID = contents of identifier/abscissa field in data row where maxima was found
NMISSING = number of non-numeric observations
TOTAL = sum of all numeric values in the field
(The following is a change in functionality starting in 2.31:)
This action can operate on one data field or several.
The data field(s) to operate on should specified in the fields attribute.
If more than one data field is specified, the data values from all the fields will be examined as
a group and the result will still be a single mean, SD, etc.
tagfield may be specified (see below) to indicate an identifier or abscissa field
which will be used to identify min and max cases (MIN_ID and MAX_ID) and may be useful in annotating
the min and/or max with a later invocation of #proc annotate.
action: total
Scan all eligible data rows, compute field total(s) and place it/them into the variable
TOTALS (see above). No new data set is created.
Field(s) to be totalled are specified in the fields attribute.
If only one field is being operated on, TOTALS will be a single numeric value.
If more than
one field is being operated on, TOTALS will be a comma-delimited
list of numeric values; individual totals may be accessed in your script
using something like the following, which would access the first
total in the list: #set T = $nmember(1,@TOTALS)
The decimal format of the total(s) is controlled by the resultformat
attribute. If total(s) are to be written in presentable notation
(a spacer for thousands, etc.) the resultformat attribute may be
preceded by a n, e.g. n%7.0f.
Other attributes
fields
dfield
list
The field(s) to be operated on. Required for any action that involves
data fields.
Example: fields: 2 5 6 7
showresults
yes | no
(alias showdata)
If yes the result data set is written to the diagnostic stream in bar-delimited format.
keepfields
dfield
list
If specified, only these fields will be kept. Other fields will be not be present in the result.
May be used with any action except where otherwise noted.
If data set field names are being used they must be manually reset using fieldnames.
Example: keepfields: 4 5 6
rejectfields
dfield
list
If specified, these fields will not be present in the result. All other fields will be retained.
May be used with any action except where otherwise noted.
If data set field names are being used they must be manually reset using fieldnames.
Example: rejectfields: 1 2 3 4
tagfield
dfield
Used with action: stats to specify an identifier or abscissa data field,
which will be used to identify min and max cases.
fieldnames
namelist
If specified, result data fields will be assigned the given names.
These names can later be used in any plotting proc to identify data fields.
namelist is a space- or comma- delimited list of names.
Names may include any alphanumeric characters with a maximum length of 38, and are case-insensitive.
Note that if field names are specified in proc getdata
and then proc processdata is used to alter the order of fields or
delete fields, then this fieldnames attribute must be
used in order to redefine the field names properly.
Example: fieldnames: date group n
complen
nchars
May be used with certain actions as noted above, to control the number of characters that are examined
when comparing or processing the field(s) of interest.
May be 50 or lower. Default is 50.
binsize
number | week | month | quarter | year | day | hour
May be used with action: count to tabulate on numeric or chronological bins.
If a number is given, numeric bins of size number are created, and numeric data are expected.
If week, month, or any of the other chronological specifications are given, chronological
bins are created and date, time, or datetime data are expected.
binmod
mid | low | high | start
Used with binsize. This determines what the presented identifiers for the bins will be.
If mid (the default) the identifier will be midrange. start is equivalent to low
but makes more sense with chronological data.
resultformat
printf-spec
Controls the decimal format of rewritten percents, accumulations, totals.
Default is %g.
Example: resultformat: %f
Example for action: totals (see above): resultformat: n%g
keepall
yes | no
May be used with accumulate, raccumulate, percent, or total.
If yes, fields that are normally replaced by newly computed values will remain in the result along
with the newly computed fields. For example, with action: percents this would allow the original values
to be present in the result along with the computed percents. Default is no.
select
selection expression
Used when action is select in order to specify the selection
condition. select can be used along with any action.
Example: select: @@4 = A
leftselect
select expression
rightselect
select expression
Used with action: join (and leftjoin and rightjoin).
Specifies selection conditions for left side of join and right side of join.
missingdatacode
string
Used with action: leftjoin and action: rightjoin to specify a missing data code to use for
filling in missing fields.
valfield
dfield
With action: summary, the data field holding the numeric value from which average (etc.) is to be computed.
outfile
filename
If specified, results are written in tab-delimited format to filename and are not retained in memory
as a new data set. This can be done to make the result available to other programs or to read in again using
proc getdata.
stack
yes | no
If no, the result data set replaces the original data set in memory (this might be desired for larger data sets).
Discontinued in version 2.40... all results are stored in memory except when outfile is specified.
More on break processing
action: breaks performs break processing, which is
useful for stepping through a sorted data set one chunk at a time and producing a plot for each chunk.
Input data must be sorted (or at least clustered ) on the break fields.
Specify fields as one or more (max 5) identifying "break" fields.
The complen attribute (see below) controls number of characters examined in comparisons.
Processing involves examining data rows one at a time, and when a difference in any of the break fields is detected
processing stops. This action is generally used within a loop as shown below.
#loop
#proc processdata
action: breaks
fields: 1 2 3
#proc endproc
#if @NRECORDS = 0
#break
#endif
#proc page
title: Account @BREAKFIELD1
#proc bars
...
#proc usedata
original: yes
#endloop
Your script can access the current contents of the break field(s) via the
BREAKFIELD1 .. n variable(s).
Your script can detect when the entire data set has been processed by checking
if NRECORDS = 0
|
