RETS Change Proposal 1: Structured Data Exchange
Author: Bruce Toback and Leo Bijnagte
Organization:
Telephone:
Address:
Email:
Status: Proposal
Date: 12/25/2001
Proposal Version: 1.0
1. Synopsis
RETS 1.0 does not provide any way for servers and clients to exchange information with internal structure except within the limited domain of a STANDARD-XML transaction. Structured Data Exchange adds a fourth data exchange type, to be designated LOCAL-XML, to allow structured information to be transmitted with a full metadata description. A LOCAL-XML-packaged query response is a well-formed XML document that can be parsed using any standard XML parser.
2. Rationale
RETS 1.0 specifies two methods for exchanging data between endpoints: COMPACT, which is a flat record format described by metadata, and STANDARD-XML, which is a structured record format that is described by a DTD and a developer note. This means that endpoints can exchange all the information for a resource but with internal structure flattened, or limited information but with a rich internal structure. The obvious weakness here is that there is no way for endpoints to exchange full information with deep internal structure.

To address this need, developers have either been forced to flatten their data structures artificially (often at some cost in completeness), or to extend the standard RETS DTD. Since such extensions are nonstandard, they inhibit the interoperability of RETS implementations that use them. Structured Data Exchange provides a standard method for describing structure, eliminating the need for nonstandard extensions when resources have a complex internal organization.

In addition, some developers have expressed the need for an upload format that uses XML rather than the present HTML field encapsulation. Structured Data Exchange provides a framework within which that need can be accomodated, although this proposal does include modification of the Upload transaction.

3. Proposal
3.1 Specification Changes

This solution involves adding a new LOCAL-XML data return type, and several new fields to the Table metadata. The LOCAL-XML return type encapsulates the returned data as XML, using <record> as the root of each returned record. The return is thus a series of <record> elements within the <DATA> section of the search transaction response.

The server describes the LOCAL-XML document using six new metadata fields:

NameTypeDescription
privateXMLTagCharacterThe XML tag or attribute name which encapsulates this field. If this element is omitted or blank, the system name is used as the XML tag.
parentFieldCharacterThe system name of the field of which this field is a subfield -- or, in other words, the system name of the structure of which this field is a member. If this element is not included in the metadata or is blank, the field is not a subfield of any other field.
isAttributeBooleanIf this metadata element is TRUE, the field's value is contained as an attribute of the parent field. If FALSE or omitted, the field is contained as an element of the parent field.
isArrayBooleanIf this metadata element is TRUE, the parent field may contain more than one of this field. If FALSE or omitted, there may be only one instance of this field within the parent. This element must be FALSE or omitted if isAttribute is TRUE.
maximumElementsIntegerThe maximum number of elements if this field is an array. If the value is zero or omitted, there is no limit to the number of repetitions. However, if isArray is FALSE or omitted, this element must either be omitted or have a value of 0 or 1.
groupClassNameCharacterA name that may be associated with this parent item. Meaningful only for fields that are designated as parent fields by one or more other fields. The value is advisory only; the client need not make use of the name provided.

This scheme permits retrieved records to be XML-encoded complete with internal structure, yet still retains the flexibility offered by having the server provide metadata. Because it generates well-formed XML, the data can be parsed by off-the-shelf tools and easily assembled into flat records, if that's what's desired. In addition, because the XML structure is specified by the server's metadata, XML can be used for upload as well.

There is no requirement that either the XMLTag or the groupClassName be meaningful or human-readable, though some sites may wish to do this. Omitting any requirement for readability means that the tag names can be very short and even machine-generated to help reduce the size of the returned data. Since a client requesting the LOCAL-XML return format is presumed to have the metadata, there is no need to present the names to an end-user.

Since the metadata now includes structure information, and since the COMPACT data format has no way of designating structures, the server must decide for any array field which single instance will be transmitted. This will normally be the most important or most significant instance, since it will be the only one seen by a user whose client requests that data be returned in COMPACT format.

This will not represent a loss in information from earlier versions of the standard, since such decisions had to be made in any case when some field in a class's data represented an array. The usual case solution in that case was to provide multiple instances of fields in the metadata (for example, PHONE_1, PHONE_2, PHONE_3 and so on); this strategy may still be used if desired.

3.2 Example of use

Assume table metadata for some resource that includes the following fragment:

SystemName Type Length XMLTag parentField isAttribute isArray maximumElements groupClassName
ListingID Char
8
ListingID
 
 
 
 
 
Commission Numeric
6
Commission
 
 
 
 
 
ListDate Date
 
ListDate
 
 
 
 
 
Address Char
0
PropertyAddress
 
F F
 
StreetAddress
StreetName Char
20
Name PropertyAddress F F
 
 
StreetNumber Char
6
Number PropertyAddress F F
 
 
ListingAgent Char
0
ListingAgent
 
 
Y
 
Member
LAName Char
40
Name ListingAgent F F
 
 
LANumber Char
10
Number ListingAgent F F
 
 
LAPhone Char
16
Phone ListingAgent F T
4
 
LAPager Char
16
PagerNumber ListingAgent F F
 
 
SellingAgent Char
0
SellingAgent
 
 
Y
 
Member
SAName Char
40
Name SellingAgent F F
 
 
SANumber Char
10
Number SellingAgent F F
 
 
SAPhone Char
16
Phone SellingAgent F T
4
 

Then a RETS client requesting LOCAL-XML with all fields selected would receive data similar the following:

    <record>
    <ListingID>18331402</ListingID>
    <Commission>6</Commission>
    <ListDate>05 DEC 2002</ListDate>
    <PropertyAddress>
        <Name>Downing</Name>
        <Number>10</Number>
    </PropertyAddress>
    <ListingAgent>
        <Name>Bill Ding</Name>
        <Number>735310487</Number>
        <Phone>888 666-1432</Phone>
        <Phone>614 234-5678</Phone>
        <PagerNumber>800 759-7243</PagerNumber>
    </ListingAgent>
    <ListingAgent>
        <Name>Rusty Nail</Name>
        <Number>638310107</Number>
        <Phone>888 555-3388</Phone>
        <Phone>604 888-5553</Phone>
        <PagerNumber>800 759-7243</PagerNumber>
    </ListingAgent>
    <SellingAgent>
        <Name>Ford Prefect</Name>
        <Number>36392837</Number>
        <Phone>877 444-3238</Phone>
        <Phone>619 888-3410</Phone>
        <PagerNumber>800 759-7243</PagerNumber>
    </SellingAgent>
    </record>

The COMPACT format for the same record would be (using "+" to indicate tab characters):

       <COLUMNS>ListingID+Commission+ListDate+Address+StreetName+StreetNumber+
               ListingAgent+LAName+LANumber+LAPhone+LAPager+SellingAgent+
               SAName+SANumber+SAPhone<COLUMNS>
      <DATA>
      18331402+6+05 DEC 2002++Downing+10++Bill Ding+735310487+888 666-1432+
         800 759-7243++Ford Prefect+877 444-3238+800 759-7243+
      </DATA>

Note that the arrays ListingAgent and Phone (within ListingAgent) have been reduced to a single instance. Although the "container" elements are explicitly transmitted in this example, they need not be. This is left as an option for the server, since container elements are empty of any textual content.

3.3. Interaction between LOCAL-XML and SELECT

Becase the SELECT argument can include any arbitrary subset of fields from the server's Table metadata, the server must modify its interpretation of SELECT when a client requests the LOCAL-XML return format. In particular, the server must include tags for all containing elements when the client requests only a contained element. For example, if a client operating with the above metadata fragment were to request only the LAPhone field in its SELECT argument, the server's response would look like:

       <record>
          <ListingAgent>
              <Phone>1-800-SELLNOW</Phone>
          </ListingAgent>
      </record>
      <record>
          <ListingAgent>
              <Phone>312 555-1212</Phone>
              <Phone>206 555-1212</Phone>
          </ListingAgent>
      </record>
     

The client must be prepared in this case to receive element names that it had not explicitly selected, since they are required to preserver the containment hierarchy.

3.4 Using Structured Data Representation for Upload

Because of the substantial changes required to modify the Upload transaction to accept an XML representation -- as well as to allow references to structured data in the valdiation language -- this proposal does not address using Structured Data Representation for upload. A separate change proposal will address this issue.

4. Development Impact
Since the functionality described in this Change Proposal is optional, there is no mandatory development impact. If a client chooses to implement Structured Data Exchange, it will need to accept and interpret the new metadata fields described in Section 3.1, and will need to incorporate an XML parser to parse the returned information. Its human-interface components, if any, will also need to be modified to accept and display multiple instances of a field, and may be changed so as to make the hierarchy explicit.

A server implementing this proposal will need to transmit the additional metadata fields, and will need substantial changes to extract and package array and contained elements.

A site using a server that implements this proposal will need to create and populate the additional metadata fields. Note that if the site is using an essentially flat data representation to begin with, the population of the additional metadata tasks is trivial, since all of the default values are acceptable.

If a site wishes to provide as much flexibility as possible to clients, it should insure that the groupClassName attributes it chooses match across all resource classes. This allows clients to implement cross-resource analysis on these entities.

5. Compatibility
A RETS 1.0 client communicating with a server that implements Structured Data Exchange will not recognize the new metadata elements and will therefore not be able to see when it has requested (via the SELECT clause) elements with internal structure. Section 3.1 of the proposal addresses this situation, requiring the server to include the most important or most visible instance of the element, together with its internal structure. This is essentially what happens now, since servers must arbitrarily flatten deep data representations in order to present a COMPACT view of the data.

A RETS 1.0 server communicating with a client that implements Structured Data Exchange will not send any of the new metadata elements. It will thus be impossible for the client to formulate a query or a data request that the server does not understand. No additional effort is required in client implementations to insure forward compatibility.

6. Proof/Need of Concept Examples
None.