A model for describing data transformations An aggregation summarizes data using aggregation functions applied to data that may be grouped by one or more variables. The resulting summary data is added to each row of the existing dataset. See the aggregation functions in the Function Library. Variables used as keys to identify groups. The expressions that compute the aggregations. An aggregation function should be used. Description of the weights to be used An expression that represents all numeric variables in the dataset, similar to `_all` in SPSS or Stata. An expression that represents all text variables in the dataset, similar to `_all` in SPSS or Stata. An expression that represents all variables in the dataset, similar to `_all` in SPSS or Stata. Describes an analysis command. An analysis command does not result in any data transformation. Combines datasets by concatenation for datasets with the same or overlapping variables. Description of files to be appended Creates a new variable identifying the source dataframe for a row with the variable name given in the value of the property Describes files used in an AppendDatasets command. Name of the file being appended. May be the name of an active dataframe. Variables to be renamed List of variables to keep List of variables to drop Logical condition for keeping rows. Logical condition for dropping rows. The software package that works with the file. The name of a file format COGS {"enumeration":["csv","txt","dat","xls","xlsx","sav","dta","sas7bdat","rds","rdata"]} Indicates whether the file format is compressed. BooleanConstantExpression takes values of TRUE and FALSE. Values of TRUE and FALSE are used in logical expressions. A collapse command summarizes data using aggregation functions applied to data that may be grouped by one or more variables. The resulting summary data is represented in a new dataset. See the collapse functions in the Function Library. The details of implementation of caseWise and columnWise exclusion vary in each statistical package. The source language is indicated in this property for reference. The name of a new, aggregated dataset to be created. Variables used as keys to identify groups. The expressions that compute the aggregations. An aggregation function should be used. Description of the weights to be used Observations are not included if any variable is missing COGS {"enumeration":["Stata"]} Variables are not included if any value in a column is missing COGS {"enumeration":["SPSS"]} CommandBase defines general properties available to all commands. The type of command Information about the source of the command. Adds a message that can be displayed with the command. Describes a source code comment. Comments should not include special characters. The text of the source code comment. A composite variable name is used to describe a variable name that is computed. An expression that evaluates to the constructed variable name. Assigns the value of an expression to a variable. The variable that is computed. The expression used to compute the value of the variable(s) Provides a name for the dataframe and an ordered list of variables (optional). Name of the dataframe. Variables in the dataframe in order An ordered list of variables used as hierarchical dimensions in a data cube or multi-index An ordered list of variables used as colum indexes in a multi-index A date-time constant expressed in ISO 8601 format. A specific instance of time in ISO 8601 format A set of commands that are performed when a logical expression is true. May also include ElseCommands to be performed if the logical expression is false. The commands in DoIf are performed once, and it expects a logical condition that applies to the entire dataframe. Use IfRows for commands that are performed on each row depending upon values on those rows. A logical expression Commands to be performed if the condition is true. Commands to be performed if the condition is false. Rows that match the selection condition are deleted from the dataset. If the logical expresion is true, the case (row) will be deleted from the working dataset. Deletes variables from the data. The variables to be dropped. This command causes the system to execute preceding commands before continuing to process the command script. The SDTL types in ExpressionBase are used in expressions, which may be numeric, text or logical. Describes the arguments in a function as specified in the SDTL Function Library. Notes: 1. The argumentName property must always be present. 2. The argumentValue property is required, and it must be a single object. When the argumentValue is a list of items in the source language, the ValueListExpression or VariableListExpression should be used. The name of the parameter. The value of the parameter. An expression evaluated by reference to the Function Library. The name of the function being called. When true, the Function property contains the name of a function from the SDTL function library. When false, the Function property contains the name of a system-specific or user-defined function. A list of parameters to the function. A group of expressions to be evaluated before expressions outside of the group. Used to control the order of operations in a formula. Expression to be evaluated. A set of commands that are performed on each row in the dataframe when a logical expression is true for that row. May also include ElseCommands to be performed if the logical expression is false. Use DoIf for a logical condition that applies to the entire dataframe and commands that are performed once. A logical expression that is evaluated separately for every row in the dataframe. Commands to be performed if the condition is true. Commands to be performed if the condition is false. InformBase includes information elements that do not transform data. Describes an invalid command. A command is invalid if it uses incorrect syntax, or is otherwise not allowed by the executing system. A message describing the issue with the invalid command. Describes an iteration process consisting of an IteratorSymbolExpression and a list of values it takes. The name used in IteratorCommands for the index variable that changes value in the loop. Describes the values that are substituted for the index variable that changes value in the loop. The name of an iterator symbol used as an index in describing the actions of a loop. A name used for the index variable that takes different values inside a loop. Rows that match the selection condition are retained in the dataset. Other rows are deleted. The logical expresion that must be true for a case to remain in the working dataset. Variables to be retained in the dataset. Variables not on the list are deleted. The variables to be kept in the dataset. Load data from a file. The name of the file to be loaded. The software package that works with the file. The name of a file format COGS {"enumeration":["csv","txt","dat","xls","xlsx","sav","dta","sas7bdat","rds","rdata"]} Indicates whether the file format is compressed. A loop creates multiple versions of a set of commands by iterating over a list of variables, numbers, or strings. Describes one or more iteration processes in this loop. Commands generated by the loop expanded by replacing tokens representing variables, numbers, or strings with their values. When TRUE, the loop has been expanded into separate commands. LoopWhile iterates over a set of commands under the control of one or more logical expressions. Since the logical conditions typically depend upon values in the data, commands executed in a LoopWhile cannot be anticipated and expanded in SDTL. Describes a condition required for the next iteration to begin. Describes a condition that ends interation. Commands within the loop. When TRUE, the loop has been expanded into separate commands. Merges datasets holding overlapping cases but different variables. The merge may be controlled by keys or grouping variables. Description of files to be merged. A variable or list of variables that acts as the unique case identifier across datasets. If MergeByVariables is absent, MergeType must be "sequential" on all files. The name of a variable set to 1 for the first row of each group of cases with the same value for the MergeByVariables variables and set to 0 for all other rows. The name of a variable set to 1 for the last row of each group of cases with the same value for the MergeByVariables variables and set to 0 for all other rows. Describes files used in a MergeData command. Name of the file being merged. May be the name of an active dataframe. Describes the type of merge performed. COGS {"enumeration":["Sequential","OneToOne","ManyToOne","OneToMany","Cartesian","Unmatched","SASmatchMerge"]} Name of a new variable indicating whether the row came from this file or a different input file. Variables to be renamed When the same variables exist in more than one dataframe. values in the "Master" dataframe can be replaced by values from a different dataframe. "Master" is the default value. "Ignore" means values in this dataframe are never used. "FillNew" is used on rows not found in the "Master" dataframe. "UpdateMissing" replaces missing values in the "Master" dataframe. "Replace" changes all values in the "Master" dataframe." COGS {"enumeration":["Master","Ignore","FillNew","UpdateMissing","Replace"]} When TRUE, generate new row when not matched to other files List of variables to keep List of variables to drop Logical condition for keeping rows. Logical condition for dropping rows. An ordered list of variables used as keys in this file to be matched to the variables in the mergeByVariables property of the MergeDatasets command. This property is only used when the key variables in this file have different names than the variable names listed in the MergeDatasets command. The software package that works with the file. The name of a file format COGS {"enumeration":["csv","txt","dat","xls","xlsx","sav","dta","sas7bdat","rds","rdata"]} Indicates whether the file format is compressed. Inserts message text in the SDTL file. Information, Warning, Error The line number of the source that the messages is related to, if relevant. The character position of the source that the message is related to, if relevant. A missing value constant. Some languages allow multiple missing value constants. The missing value as it appears in the system (e.g., .a, .b, .c). If Null, this is the system missing value. The NewDataframe command copies or creates a new dataframe. It can be used in two ways. An existing dataframe can be copied to a new dataframe by using the consumesDataframe and producesDataframe properties of NewDataframe. The new dataframe will be a "deep" copy in the sense used in R and Python. NewDataframe can also be used to create an empty dataframe of a specific size. In Stata, the "set obs #" command will create a dataframe with a user-defined number of rows. All values are assumed to be missing. This may be used in simulations to preset a number of simulated observations, which are then filled with randomly generated data. Number of rows in new dataframe Number of columns in new dataframe NoTransformOp is used for a command in the original script that provides important information but does not have a function in SDTL. For example, “library()” in R loads a package of R functions. Since the Parser detects the library, the SDTL will reflect the library that is used, and commands derived from the library will be translated in the SDTL script. However, it is useful to know which library is active for auditing the R script, even if it does not perform any data transformations. Defines a range of numeric values. Starting value for range Ending value for range Increment for stepping through range A numeric constant. A number Type of the number COGS {"enumeration":["Integer","Real"]} Represents the largest numeric value supported by a system. Represents the smallest numeric value supported by a system. The program represents a list of commands from a statistical package. ID of the object being referenced. COGS:identification=true;position=0 COGS identification(position=0) The name of the file containing the source code. The language of the source code. The MD5 hash of the contents of the file. The SHA-1 hash of the contents of the file. The date and time the file was last updated. The size of the file in bytes. The number of lines in the source file. The number of commands detected in the source file. Messages related to the parsing of the source file. The name of the parser used to generate the SDTL. The version of the parser used to generate the SDTL. The version of the SDTL model. The date and time the SDTL was generated. The list of commands that make up the program. Describes recoding values in one or more variables according to a specified mapping. The Recode command can either describe a recoding of one or more individual variables, or a range of variables. When one or more individual variables are described, a new variable name can be specified. In this case, the original variable is left alone, and a new variable is created with the recoded values. The variables that will have their values recoded. The resulting values may be either stored in the same variable, or a newly created destination variable A range of variables to which the recode rules are applied. The resulting values are stored in the same variable. A mapping describing which values will be recoded to which new values Describes how values will be recoded. The values to be recoded. The new value A value label for the new recoded value, if appropriate Describes a variable that will have its values recoded. The name of the variable which will have its values recoded The name of the new variable into which the recoded values are inserted. This may be the same as the source variable if values are recoded in place. Rename changes the names of variables. A list of variable rename objects, which includes the old name and the new name. Variable names before and after a variable is renamed. The old name of the variable. The new name of the variable. Descripbes a new variable created by reshaping a dataset from wide to long or vice versa. Name of new variable created by this command. Label for new variable created by this command. Source variables in the original dataset used to create this variable. The stub is a string used in the names of variables in the wide dataset. A list of values that produce new rows (long) or columns (wide) for this variable. Name of the variable that will contain the value of the index for this row. Label for the variable in IndexVariableName. Creates a new dataset with multiple rows per case by assigning a set of variables in the original dataset to a single variable in the new dataset. New variables created by this command. New variable identifying the case number in the wide data that created this row. One or more variables identifying unique rows in the wide data. Variables to be dropped from the new dataset. Variables to be kept in the new dataset. When set to TRUE, rows in which all constructed variables are missing are not deleted. New variable with the number of cases in the long dataset that were created from the source row in the wide dataset. Label for the CountByID variable. ReshapeWide is not supported in the current version of SDTL, because it depends on values in the data. However, it may be useful when values of the index variable are available in the metadata file or the data can be processed. New variables created by this command. One or more variables identifying unique rows in the wide data. Variables to be dropped from the new dataset. Variables to be kept in the new dataset. Writes a dataset to a file. The name of the file to be saved. The software package that works with the file. The name of a file format COGS {"enumeration":["csv","txt","dat","xls","xlsx","sav","dta","sas7bdat","rds","rdata"]} Indicates whether the file format is compressed. Sets the data type of a variable. Examples: ``` "dataType": "Text", "subTypeSchema": "Stata 15.1", "subType": "str1" ``` ``` "dataType": "Numeric", "subTypeSchema": "https://www.ddialliance.org/Specification/DDI-CV/DataType_1.0.html" , "subType": "Int" ``` The variables that will have their type set General type of a variable, e.g. "Text" or "Numeric". COGS {"enumeration":["Text","Numeric","Boolean","Date-Time","Factor"]} A vendor or standards body with a controlled vocabulary. The value can be a URL. The name used in the associated schema. Changes a property of a dataframe. The name of the property to be set (for example, 'Title' or 'Subtitle'). The value of the property. Sets the display or output format for a variable. Examples: ``` "displayFormatSchema": "SAS" "displayFormatName": "DOLLAR6.2" ``` ``` "displayFormatSchema": "Stata 15.1" "displayFormatName": "%tcDDmonCCYY_HH:MM:SS" ``` The variables that will have their format set A vendor or standards body with a controlled vocabulary. The value can be a URL. The name used in the associated schema. Defines values that are treated as missing values for a list of variables. The list of variables to which the missing values are assigned The values to be considered as missing values Describes the assignment of labels to categorical values. The variables to which a label will be assigned The label to be assigned to the variables Describes the assignment of a label to a variable. The name of the variable to which a label will be assigned The label to be assigned to the variable Sorts rows in the dataframe in a specified order. Describes how cases are sorted. Describes a criterion by which cases are sorted, including the variable name and whether to sort ascending or descending. The variable used to sort. The direction in which to sort. COGS {"enumeration":["Ascending","Descending"]} SourceInformation defines information about the original source of a data transform. The line number of the beginning of the transform code The line number of the end of the transform code The character index of the beginning of the transform code The character index of the end of the transform code The original source code of the data transform code The source code of the data transform code after processing macros or loops A text string. A string Defines a range of string values, such as "A to N". Starting value for range Ending value for range A time duration constant expressed in ISO 8601 format. A duration of time in ISO 8601 format TransformBase defines properties available for transform commands. Signify the dataframe which this transform produces. Signify the dataframe which this transform acts upon. Represents any values not previously handled (for example, in a set of recode rules). Describes an unsupported command. An unsupported command is valid syntax, but not supported by the parsing application. Associates a label with a value in a categorical variable. The value to which a label is assigned The label to be assigned to the value Wraps a list of other expressions. The list of expressions. A list of variables which may include variable names (VariableSymbolExpression, CompositeVariableNameExpression), variable ranges (VariableRangeExpression), and variable lists. An array of variables and/or variable ranges A list of variables in adjacent columns defined by the variable names of first and last columns. Lowest variable name in range Highest variable name in range Types used to describe variables in expressions. A reference to a variable. The name of a variable Weight describes the type of weight to apply and variables holding weights. The variable used as a weight in the operation. Type of weight to use COGS {"enumeration":["frequency","probability","Stata_aweight","Stata_iweight"]}