RETS Change Proposal 8: Query Language (Search Syntax) Modifications
Author: Leendert M. "Leo" Bijnagte
Organization: Fidelity National Information Solutions
Telephone: (612)661-1087
Address: 100 Washington Ave. South #900, Minneapolis, MN 55401
Email: leob@vistainfo.com
Status: Proposal
Date: 03/18/2002
Proposal Version: 1.0
1. Synopsis
Changes are proposed to the Query language to clarify usage and add some missing functionality.
2. Rationale
The specification of the syntax for searches defined in version 1.0 of the spec is not only confusing but in spots inaccurate. Due to the omission of a NOT operator on query components,  the current DMQL language does not allow for the generation of certain types of queries commonly used by clients and supported by servers. Usage of nesting and the AND and OR operators is shown in the examples but the production rules are not clear in the augmented BNF. The suggested revisions (tentatively DMQL2) are to the augmented BNF of the Boolean elements of the language and extensions to the definitions that make up field-criteria.

The suggested revision for defining the Query takes its form from that of the <search condition> found in the BNF of SQL92 (www.contrib.andrew.cmu.edu/~shadow/sql/sql1992.txt) with the exception that the intermediate production <boolean test> and its <truth value> have been deprecated to always TRUE to retain compatibility for existing queries.

Separate but included revisions to the details of the field-criteria are three-fold. First, the addition of string-literal to field-value to provide searching for string field content that might include non-ALPHANUM characters. The string-literal definition matches that of SQL-92. Second, replacing the 1*DIGIT production to the newly defined  number to support decimal numbers.  Third, a change to the definition of range to use number and add string-eq to permit ranges on all datatypes to avoid forcing complex query construction for the currently excluded datatypes.

With the use of the specified revisions, field-criteria becomes a subset of the SQL92 <predicate> options that 
provides equivalent syntax for basic <predicate comparison>, <predicate between>, and <predicate in> expressions without 
the need (or ability) to specify joins or sub-queries. This fits well with the flattened views embodied in the Resources and the way Metadata links Lookups to Fields. 

3. Proposal
3.1 Specification Changes

1. A revised BNF for the Query language and instructions about string-char and number substitution in section 7.7:

Query ::= search-condition

search-condition::== query-clause | ( search-condition or query-clause)

query-clause ::= boolean-element | ( query-clause and boolean-element )

boolean-element :: = [not] query-element

query-element ::= field-criteria | ( "(" search-condition ")" )

or ::= "OR" | "|"

and ::= "AND" | ","

not ::= "NOT" | "~"

field-criteria ::= "(" field "=" field-content")"

field-value ::= lookup-list | string-list | range-list | period | number | string-literal

lookup-list ::= lookup-or | lookup-not | lookup-and

lookup-or ::= "|" lookup *( "," lookup )

lookup-not ::= "~" lookup *( "," lookup )

lookup-and ::= "+" lookup *( "," lookup )

lookup ::= <any legal ALPHANUM value for the field as defined in the metadata>

string-list ::= 1*( string *( "," string ))

string ::= string-eq | string-start | string-contains | string-char

string-eq ::= 1*ALPHANUM

string-start ::= 1*ALPHANUM "*"

string-contains ::= "*" 1*ALPHANUM "*"

string-char ::= *ALPHANUM 1*"?" *ALPHANUM

string_literal ::= <"> 1*( *( PLAINTEXT except <"> ) *( 2<"> ) *( PLAINTEXT except <"> ) ) <">

range-list ::= 1*( range *( "," range ))

range ::= between | greater | less

between ::= ( period | number | string-eq ) "-" ( period  | number | string-eq )

greater ::= ( period | number | string-eq ) "+"

less ::= ( period | number | string-eq ) "-"

period ::= (date | datetime | time)

number ::= 1*DIGIT ["." *DIGIT]

date ::= (year "-" month "-" day) | TODAY

datetime ::= (year "-" month "-" day "T" hour ":" minute":" second ["."fraction]) | NOW

time ::= (hour ":" minute ":" second ["." fraction])

fraction ::= 1*3DIGIT

second ::= 2DIGIT

minute ::= 2DIGIT

hour ::= 2DIGIT

day ::= 2DIGIT

month ::= 2DIGIT

year ::= 4DIGIT

NOW ::= "NOW" <the host should substitute this with its current date/time>

TODAY ::= "TODAY" <the host should substitute this with its current date>

In processing a literal string, a server MAY substitute a string-char expression (?s) for the range of characters that contain any non-ALPHANUM not supported by that server.

In processing decimal numbers, a server MAY round as follows, down for the bottom of ranges or values less than .5 and round up for the top of ranges or values .5 or greater.. 

2. The addition of DMQL2 as a Query Type in section 7.3.2:

QueryType       ::= DMQL | DMQL2 

An enumeration giving the language in which the query is presented. The valid value are "DMQL" for RETS/1.0 compliance and "DMQL2" which indicates the query language described in Section 7.7.

3.2 Implementation Notes

This proposal minimizes the changes to the query syntax while filling some important holes. The author hopes to have balanced the implementation effort that will be required by servers with the need for more SQL like syntax for the clients.

4. Development Impact
The  proposal requires server's search mechanisms to support the additional functionality of the NOT operator. If such functionality is not currently available, servers would need to provide at minimum a way to break down a DMQL2 query into multiple DMQL queries and perform Boolean logic on the result sets to be compliant with the new version.

The proposal  for string literal support includes an option for servers that could limit the impact to parsing  for quote characters and doing ? substitution for some range of characters.

The proposal for allowing decimals in numbers includes an option for servers to do rounding that could limit the impact to parsing for fractional elements and rounding. 

The proposal for including string-eq in ranges will require servers that do not have grater than or less than search functionality on character data to provide such functionality.

Clients will not be required to change in any way to be compliant.

5. Compatibility
Existing DMQL client query syntax would continue to conform while servers would need to provide processing for the new version to be compatible with the new release of the spec. Servers could elect to remain at version 1.0 of the spec; clients wanting to make use of the new syntax may want to retain some accommodation for connecting to servers at version 1.0 of the specification.
6. Proof/Need of Concept Examples
None.