RETS Change Proposal 32: Distributed Data Base Enhancements
Author: Leo Bijnagte
Organization: Fidelity National Information Systems
Telephone: (612)661-1087
Address:
Email: leob@fnis.com
Status: Proposal
Date: 03/31/2003
Proposal Version: 1.0
1. Synopsis
This proposal adds functionality to RETS to permit clients to maintain local copies of a data base efficiently and accurately with minimal impact on the server. It includes changes to metadata, an enhancement to the Search transaction to facilitate returning data in manageable chunks, and a new facility for querying server management parameters.
2. Rationale
RETS currently lacks some key elements necessary to perform a common function: replication of selected parts of a server's data. This is currently possible in RETS using tools such as modification timestamps, the search transaction Limit and Offset parameters, METADATA-FOREIGN_KEYS and so on. However, replication using these tools is imperfect and often requres special arrangements between endpoints. This is contrary to the goal of RETS, which is to obviate the need for such special point-to-point arrangements.

The proposal requires changes to the standard in three areas. First, the RETS mechanism for dealing with limits on the number of returned records is modified so that it's possible to retrieve data sets larger than the general search site limit without causing a significant server load. This is done by putting the server in control of the chunking mechanism and chunked transfers. There is also a mechanism that allows a client to query a server as to the minimum permissible time between updates, and the server is provided with a mechanism for informing the client when those criteria have been violated. This allows the client to adjust its behavior for the most efficient synchronization schedule.

Second, the metadata is enhanced in four ways: Flags are added to allow the client to detect which fields are affected by which timestamps. Cross-references are added in the Table, Object, and Foreign Key metadata to facilitate retrieval of child records whose data were previously stored from the flattened representation of the main record.

Finally, the proposal adds a new transaction, called GetSettings. This transaction returns a dictionary of server-specific settings in somewhat the same manner as the GetMetadata transaction. Its initial intended use is to allow a client to query the server's throttling (bandwidth control) parameters, but it is envisioned that there will be many other uses in the future. These could include querying specific server capabilities rather than forcing a client to infer this from the version number and trial-and-error transactions.

3. Proposal
3.1. Search Transaction Changes

In Section 7.4 Optional Request Arguments, add the following:

Key = ".EMPTY." | NextKeyValue

If the client provides no value for Key, the search proceeds as per the current specification.

If it is set to .EMPTY. or a NextKeyValue and the request meets the serverŐs requirements for KEY support as defined in the TABLE-METADATA for the Resource and Class, the server MUST provide a <NEXTKEY> tag with a NextKeyValue if it sends a <MAXROWS> tag in the response. For Resources deemed replicable, servers SHOULD support either KeyLimit=NONE in settings or have the resource's Keyfield as KeySelect.

If it is set to NextKeyValue, the server MUST respond with a result set that starts at an offset 1 record past the previous request and guarantees the request chain that started with an .EMPTY. value will not miss any records that still meet the query parameters. The results set MAY contain records that have been additionally modified since the beginning of a request chain that started with a Key=.EMPTY. argument. A NextKeyValue is opaque to the client, only valid once and only in the context of a single request chain. The request chain ends when the Server does not include a <MAXROWS> tag in the response. At the boundary of its limits, a server MAY send <MAXROWS> and <NEXTKEY> in a response and not deliver any records to the subsequent request (ReplyCode 20201), likewise ending the request chain.

If the client includes the Key request argument with a NextKeyValue rather than .EMPTY., the client MUST send all the other request arguments and values used in the original request. If the client drops arguments or includes different values, the server's response MAY no longer be reliable. If a client issues a search for a given resource and class that does not include the Key optional request argument or for another resource or class, the NextKeyValue becomes invalid and can not be used again.

In Section 7.6, the Response Body format is modified to add the following:

<next-key-tag>

immediately before the <RETS-REPLY> tag. The next-key-tag> is of the form

<NEXTKEY>NextKeyValue</NEXTKEY>

where NextKeyValue is an opaque string not exceeding 64 characters that may be used in the next transaction in this session as an offset to step through the result set of the query.

In addition, three new response codes are defined.

Reply Code
Meaning
20212Invalid Key Request
The transaction does not meet the server's requirements for the use of the Key option.
20213Invalid Key
The transaction uses a Key that is incorrect or is no longer valid.

3.2 Metadata Changes
3.2.1. METADATA-TABLE
Field Name
Content Type
Description
ModTimeStampBooleanWhen true, indicates that changes to this field update the class's ModTimeStamp.
ForeignKeyCharacterWhen nonblank, indicates that this field is normally populated via a foreign key. The value is the ForeignKeyID from the METADATA-FOREIGN_KEYS table.
ForeignFieldCharacterThe SystemName from the child record accessed via the specified ForeignKey.
KeyQueryBooleanWhen true, indicates that this field may be included in a query that uses the Key optional argument.
KeySelectBooleanWhen true, indicates that this field may be included in the Select list of a query that uses the Key optional argument.

3.2.2. METADATA-CLASS
Field Name
Content Type
Description
ClassTimeStampCharacterThe SystemName of the field in the METADATA-TABLE that acts as the timestamp for this class.
DeletedFlagFieldCharacterThe SystemName of the field in the METADATA-TABLE that indicates that the record is logically deleted. If this field is specified, then DeletedFlagValue must be specified as well.
DeletedFlagValueCharacterThe value of the field designated by DeletedFlagField indicating that a record has been logically deleted. If the type of the field named in DeletedFlagField is numeric, then this value is converted to numeric type before comparing. If the type of the field named in DeletedFlagField is character, then the shorter of the two values is padded with blanks and the comparison made for equal length.

3.2.3. METADATA-OBJECT
Field Name
Content Type
Description
ObjectTimeStampCharacterThe SystemName of the field in the METADATA-TABLE that acts as the timestamp for objects of this type. This SystemName MUST appear in all classes which have objects of this type.
ObjectCountCharacterThe SystemName of the field in the METADATA-TABLE that contains the count of objects of this type. This SystemName MUST appear in all classes which have objects of this type.

3.2.4. METADATA-FOREIGN_KEY
Field Name
Content Type
Description
ConditionalParentFieldCharacterThe SystemName of a field in the parent's METADATA-TABLE that should be examined to determine whether this parent-child relationship should be used. If this is blank, the relationship is unconditional.
ConditionalParentValueCharacterThe value of the field designated by ConditionalParentField indicating that this relation should be used. If the type of the field named in ConditionalParentField is numeric, then this value is converted to numeric type before comparing. If the type of the field named in ConditionalParentField is character, then the shorter of the two values is padded with blanks and the comparison made for equal length.

To determine the correct child relationship for a parent where there are multiple instances of the given ParentResourceID/ParentClassID/ParentSystemName, the client compares the value of the ConditionalParentField to the value given by ConditionalParentValue, and uses the first foreign key for which the values match according to the rules given in the table.

3.3. New ServerInformation transaction

This change proposal introduces an extensible mechanism for obtaining server settings. The settings may be system-wide, or may be associated with a specific resource and class.

3.3.1. Required Request Arguments

There are no required request arguments. A GetSettings transaction with no arguments returns system-wide settings.

3.3.2. Optional Request Arguments
ResourceThe name of the resource for which settings are desired. A value of "*" indicates that settings for all resources should be returned.
ClassThe name of the class for which settings are desired. A value of "*" indicates that settings for all resources should be returned. If the value of Resource is "*", then Class MUST be "*" or omitted (in which case, the asterisk is implied).

3.3.3. Response

The response is a well-formed XML document of the following form:

    <ServerInformation>
       <Parameter name="
parametername" [resource="resource ID" [class="ClassID"value</Parameter>
       .
       .
       .
   <ServerInformation>

The server MUST supply the information that applies to the Class level even if the information is global to the system. That is, the client is not required to infer information from the class hierarchy.

The well-known names for parameters are given below. A server extending this specification with additional parameter names MUST precede the parameter name with the string "X-".

Parameter
Level
Type
Description
CurrentTimeStampSystemDateTimeThe current system date and time, including time zone, in ISO 8601 format.
LastTimeStampResourceClassDateTimeThe most recent modification timestamp of any record in the class, in ISO 8601 format.
MinimumLimitResourceClassNumeric/StringThe minimum Limit value for any search in this class. The value NONE may be returned if there is no minimum limit.
KeyLimitResourceClassNumeric/StringThe minimum Limit value for any search in this class that includes a Key optional parameter. The value NONE may be returned if there is no minimum limit.
ReplicationSupportResourceClassCharacterAn indication of the level of replication support available for the given resource/class:

NReplication is not supported for this resource/class: there are fields for which it is not possible to determine the last change date, or there is no support for the Key optional search argument.
YAll fields are marked as to their controlling timestamp or foreign key, and the server supports the Key optional search argument. A blank query may be used to retrieve all records that the current user is permitted to access.

.

KAll fields are marked as to their controlling timestamp or foreign key, and the server supports the Key optional search argument. A query MUST contain one or more of the fields marked in the metadata with the KeyQuery flag.

.


3.3.4. Response codes
Reply Code
Meaning
20601Not supported
The transaction is not supported for the given resource/class.
20602Miscellaneous error
The transaction could not be completed; the ReplyText contains additional information.

4. Development Impact
Clients that wish to do accurate database replication will need to implement the optional Key argument logic. Note that this is a new capability, so there are no compatibility issues.

The update has a moderate impact on servers that choose to implement the functionality. In addition, the new metadata for supporting the Key function may be time-consuming to generate.

The addition of the ServerInformation transaction represents a mandatory enhancement for 1.6-compatible servers.

5. Compatibility
This change is backward-compatible with respect to clients: no current transactions are altered by this proposal.
6. Proof/Need of Concept Examples
None.