Transform

The Transform section of the Xml document contains several important settings.
Within the Datasets element there are a series of Dataset elements that define the transformation and QA settings for this particular configuration. One example is for a BuildingInteriorSpace as shown below:

<Dataset name=”BuildingInteriorSpace” qa=”CheckFields,RepairGeometry,CheckGeometry” sourceIDField=”Handle” sourceNameField=”DocName”>

  • “name” is the name of the feature class or table in the transformation database
  • ”qa” contains options for the types of QA to perform on the Dataset
    • CheckFields indicates whether fields should be checked as defined in the Fields documentation.
    • RepairGeometry will run the ArcGIS RepairGeometry tool to look for issues in the source data that may prevent loading to an Enterprise Geodatabase. This tool will run up to 3 times to try to fix geometry issues and will only report an error if the 3rd time does not fix all geometry issues.
    • CheckGeometry will run the ArcGIS CheckGeometry tool to look for errors in the Geometry. In the example above we want to run it after the repair operations, but it can also be run without Repair if you do not want the tools to touch the original geometries.
    • “sourceIDField” is the name of the field in the source dataset that you want to use to log in error reporting.
    • “sourceNameField” is the name of the source transaction or document you want to use for error logging.

The next section contains a list of Fields that are part of the transformation. They generally contain the name of the Source field, the name of the Target field, and the method used to transform from the source values to target values. A partial example is shown below for a road centerline dataset:

<Dataset name=”Roads” qa=”CheckFields,CheckGeometry” sourceIDField=”ROADSEGMENTID” sourceNameField=”STREET_NAME”>
<Field>
<SourceName qa=”Required,Unique”>ROADSEGMENTID</SourceName>
<TargetName qa=”Required,Unique”>GeoID</TargetName>
<Method>Copy</Method>
<FieldProperties>LONG</FieldProperties>
</Field>
<Field>
<SourceName qa=”Required”>STREET_NAME</SourceName>
<TargetName qa=”Required”>FullStreetName</TargetName>
<Method>PythonCalculate</Method>
<PythonExpression>”|STREET_NAME|”.title()</PythonExpression>
<FieldProperties>TEXT,,,200</FieldProperties>
</Field>
</Dataset>

There are 5 main Methods for performing the value transformations:

    • DefaultValue – set the target field to this value.

 <Field>
<SourceName qa=”Required”>MUNICIPALITY</SourceName>
<TargetName qa=”Required”>Municipality</TargetName>
<Method>DefaultValue</Method>
<DefaultValue>Waterloo </DefaultValue>
<FieldType>TEXT</FieldType>
<FieldLength>30</FieldLength>
</Field>

The example above shows how to map a source field named “MUNICIPALITY” to the target field “Municipality”, and sets the default value for each of those fields to “Waterloo”. Notice the field properties section defines the type and maximum length of input required for the target name. (i.e., mapping to a value larger than 30 characters would cause an error)

    • Copy – copy the value of the Source field to the field with the Target name.

 <Field>
<SourceName qa=”Optional”>EDIT_DATE</SourceName>
<TargetName qa=”Check”>LastModified</TargetName>
<Method>Copy</Method>
<FieldType>DATE</FieldType>
<FieldLength>8</FieldLength>
</Field>

This example shows how the date values (mm/dd/yyyy) from the source field to the target field “LastModified”. More info is provided on source/target qa options beneath this section.

    • ValueMap – the values in the Source field need to be mapped to a similar list of different values. For example, the Source field may contain “CAMB” and the target value should be “Cambridge”.

 <Field>
<SourceName qa=”Optional”>JURISDICTION</SourceName>
<TargetName qa=”ValueMaps”>OwnedBy</TargetName>
<Method>ValueMap</Method>
<ValueMapName>Ownership</ValueMapName>
<FieldType>TEXT</FieldType>
<FieldLength>35</FieldLength>
</Field>

A corresponding section for <ValueMaps> must also be added to the code after your <datasets>, with the options for the value map set as follows for current example:

 <ValueMaps>
<ValueMap name=”Ownership”>
<SourceValues>CAMB,KIT</SourceValues>
<TargetValues>Cambridge,Kitchener</TargetValues>
</ValueMap>
</ValueMaps>

    • PythonCalculate – enter any python expression here to calculate the new value. For example, to create a new value for building+floor we could use the expression: “|BUILDING|” + “-” + “|FLOOR|”. Use ! or | on either side of attribute values, do not use ! in any other way in expressions unless you use | to delimit the field names.

 <Field>
<SourceName qa=”Required”>BLDG</SourceName>
<TargetName qa=”Required”>BUILDING_KEY</TargetName>
<Method>PythonCalculate</Method>
<PythonExpression>”|BUILDING|” + “-” + “|FLOOR|”</PythonExpression>
<FieldType>TEXT</FieldType>
<FieldLength>100</FieldLength>
</Field>

You can write your own python calculate functions in your arcpy/myGizinta.py file. This is already included by the AddFieldsAndValues.py script and gives you the flexibility to write just about anything for the expresession.
<PythonExpression>myFunction(‘test this out’)</PythonExpression>

    • Otherwise – you can use an Otherwise value to store a value for anything that doesn’t match the set of value maps. For example, a number of SpaceType values from the source should just be set to “Assignable” in the example below

 <Field>
<SourceName qa=”Required”>SPACE_TYPE</SourceName>
<TargetName qa=”Required”>SpaceType</TargetName>
<Method>ValueMap</Method>
<ValueMapName>SpaceTypeMap</ValueMapName>
<ValueMapOtherwise>Assignable</ValueMapOtherwise>
<FieldType>TEXT</FieldType>
<FieldLength>30</FieldLength>
</Field> …

<ValueMaps>
<ValueMap name=”SpaceTypeMap”>
<SourceValues>WASH,KIT,HALL</SourceValues>
<TargetValues>Washroom,Kitchen,Hallway</TargetValues>
</ValueMap>
</ValueMaps>

For example, a number of SpaceType values from the source should just be set to “Assignable” in the example above.

    • None – in this case the source and target field names are the same so no transformation is required. This option is included because you may still want to include field QA checks on those fields.

 <Field>
<SourceName qa=”Required”>AddressID</SourceName>
<TargetName qa=”Required”>AddressID</TargetName>
<Method>None</Method>
<FieldType>TEXT</FieldType>
<FieldLength>30</FieldLength>
</Field>

The final element in the Field elements is FieldType. FieldType can contain any valid parameters that would be passed to the ArcGIS AddField tool. Refer to the ArcGIS Tool help for more details on the syntax.
In each of the Fields there is also a set of attributes for both the Source and Target fields for QA settings. Generally speaking these can be

  • “Required” – which will check to make sure there are values present (not null), and
  • “Optional” – which will not check anything.

There are also a few other special cases:

  • Unique – ensure the field values are unique
  • ValueMaps – check that values match the ValueMap values
  • Check – check the field QA properties but don’t report errors. This is similar to “Required” with no errors thrown but issues will be logged.