RETS Working Group Meeting: December 9-10, 2003


Search

December 9

This meeting had simultaneous tracks for a portion of both days; see the agenda in the meeting package for details. These notes do not necessarily reflect the chronological order of the meeting.

Workgroups

Data Dictionary

The Data Dictionary workgroup had two primary agenda items: to discuss the use of the proposed submission template, now that two submitters had some experience with it, and to discuss the submissions themselves. However, we didn't get quite that far.

The template, as revised before the most recent conference call, was deemed adequate.

The discussion of items started out innocently enough, but quickly became bogged down in structural issues. The main issue was over the inclusion of units as an attribute. It was suggested that a standard set of units be developed, but that was only one issue. One participant asked why the system of units (English or metric) couldn't be specified at a high l level in the DTD, thereby eliminating the need for separate units fields on each measurement. It was pointed out that simply specifying a system wasn't sufficient both because different units within the system could be chosen for measurements (e.g., either square feet or acres for lot size), and because there are some locations where systems are mixed. It was agreed that a conference call agenda item would be posted to deal with the issue of creating a standard list of measures.

Further discussion of the initial submission by HomeStore dealt with the issue of structure. It appears that it is very difficult to separate the idea of annotations for metadata -- pure standard names -- and issue of the eventual DTD structure in which these new fields will reside. Considerable time was spent discussing the structure -- so much so, that we never finished discussing the submission. It was agreed that a conference call would be scheduled to discuss the submission.

Another issue that came up was the idea that if a standard name existed for a datum, servers would be required to provide that datum. No amount of discussion seemed to quell this fear, although it was pointed out by several participants that if we follow this line, we can never propose any new fields. In the end, the group decided to continue with the additions.

The issue of standard name compliance was also discussed. There was a question of why everyone had to supply all standard names, though that was never a requirement. The group also discussed the idea of granting a "level" of standard-name compliance. There wasn't much enthusiasm for this until Paul Stusiak proposed that the compliance not be based on number of standard names, but on having all standard names required for a given functional area. This could also be combined with having other things required for that functional area: specific transactions and resources.

This plan was generally adopted, and the first three functional areas were proposed: HomeStore (perhaps called "aggregation"? -bt), CMA and distributed data base. Leo, Dave Sullivan and Steve Verba will work on the CMA functional area requirements specification; Steve White and Art will work on distributed data base, and HomeStore will work on Aggregation. The group will ask Compliance to develop tests for these functional areas.

Compliance

The compliance agenda was concerned primarily with the addition of new tests to the existing server and client compliance testers.

The first issue was whether the client compliance tester would continue to be hosted at NAR's Center for Realtor Technology (CRT) or would move to a third party, in this case, Avantia. Steve Verba, from Avantia, mentioned that it would be harder to come up with a fixed price for doing client testing (the fixed price for server testing is $500) because of the amount of variability in client capabilities. He said he would take the question back to the company and provide an answer shortly.

It was agreed that the existing tool should be left on-line, since it is a valuable service even though security issues prevent it from being used for actual final certification. Final certification would be done either through the trusted third party, or through the plugfest, as now.

Workflow is an issue with the current version of the compliance checker. The current compliance checker require logging in and logging out several times in order to verify that various login challenges are handled correctly. The problem with this is that vendors who have embedded clients that function under automatic control as part of a larger system may not have a way of driving their client software through that particular sequence of operations. It was suggested that the client compliance tester be changed to be somewhat more passive, recording activity and evaluating it rather than being driven through a prespecified sequence of operations. CRT said that this might be possible for some cases, and agreed to see what it would take to do this job.

In response to a question from a prospective client vendor, Bruce Toback asked the group for a statement on whether a program that used the current VB control DLL would automatically be considered compliant. The response, after a discussion by Avantia on the exact capabilities of the DLL, was that it would not: it is possible to use the current DLL in a non-compliant manner, as it only implements low-level communication and has no semantic knowledge. It might be possible to enhance the DLL so that it could have complete control over the protocol, but it doesn't work that way now.

The group evaluated a long list of additional server tests proposed by Avantia and others. Most of these had to do with "negative" testing, insuring that the server under test produces predictable responses to incorrect transactions. It was noted by several people that there are really very few places in the specification where a particular error response is mandated, although there are plenty of very specific error codes available. We need to tighten up the specification before we can do too much negative testing for pass/fail, but we can provide some guidance within the compliance tester.

The list of suggested additional tests is available as a separate list.

Finally, the Compliance workgroup discussed the possibility of doing functional-area testing, as suggested by the Data Dictionary workgroup. Steve Verba stated that the compliance checker could be enhanced to cover this possibility. (See the Data Dictionary workgroup notes for additional information on this suggestion.)

Marketing

Media

Main Meeting

The main meeting opened with a recap of the workgroups by the workgroup leaders. See above for notes from the workgroups.

Presentation by American Management Systems

The second item on the agenda was a presentation by Greg Stanton of American Management Systems (AMS), a consulting company analyzing security in the real estate industry for NAR. The purpose of the presentation was to acquaint the participants with some of the security issues that could be or that will need to be addressed by RETS.

While the presentation was short on detail, the main thrust was that a federated identity model will become important for the real estate industry as it moves into sharing and tracking that data that is collected by agents, including listing information. The slides showed a system that used a single security framework connecting to a number of identity providers and providing authentication for a number of different services. The next generation of RETS may incorporate some of these features.

Introduction to 2004 Priorities

Myron Adams presented the list of RETS priorities for 2004. The document is available separately. The purpose of the presentation was to serve as the seed for a discussion to take place Wednesday morning.

Demonstrating one of the items on the priority list, Steve Verba showed the work-in-progress on the RETS "sandbox", a shared development environment that can contain buildable versions of RETS open-source code.

December 10

Implementation Forum

Kevin McQueen led an implementation forum for participants who were not working on standards maintenance. His extensive notes detail the issues that were raised and some of the proposed solutions.

Discussion on 2004 Priorities and Meeting Logistics

There was surprisingly little discussion on the 2004 priorities presented the previous afternoon. Such discussion as there was involved mostly clarification of items on the list, rather than suggestions for changes.

A number of suggestions were made for action at future meetings:

Stuart Schuessler suggested that we add a forum for MLS operators. This might include a training component as well as a discussion of implementation issues. He noted that MarketLinx sponsored such an event for some of its customers and the event was viewed as very valuable by its participants.
Stuart also suggested that we add implementations in languages other than Java. It was pointed out that there is a Visual Basic implementation, but Stuart would prefer to see a C++ or C# implementation.
Daniel Bascilica suggested that the client compliance tester be augmented with test data that can actually be used to verify the output. Currently, it's not clear what output a client should produce, beyond simply complying with the low-level protocol elements.
Noting the element in the 2004 priorities list for creating a "generalized RETS client", Kristin Carr noted that such a facility should be a commercial product, rather than a "put it out there" project. She stated that brokers really want to have someone they can call for support, which isn't possible with an open-source unsupported product. This was echoed by several other participants.

Procedural Issues

New Change Proposal Path

Due to the large and increasing size of the group -- this meeting had almost 60 participants -- it was suggested that change proposals now go through one of the existing workgroups for evaluation an discussion, and that change proposals then be presented to the full group with a firm recommendation for passage or rejection. Change proposals could still be presented and acted on by the full group, but this would be the exception and would be done only when the proposal did not fall into the purview of one of the workgroups (or as an appeal when a workgroup recommends rejection.

This proposal, as amended, passed unanimously.

Requirement for Implementation

Bruce Toback proposed that RETS adopt a procedural change that parallels other standards groups such as the World Wide Web Consortium. This requires that any change proposal not be adopted for the final version of a standard until there were at least two working implementations of it. The idea is to provide a practicality filter to insure that any change proposal is implementable, and that implementation experience can be fed back into the wording of the standard.

Stuart Schuessler objected, saying that requiring two implementations was too onerous, and that only one should be required. He stated that there might not be more than one vendor who wants to implement, so requiring two would mean that some proposals never make it into the standard. The group agreed that one of the two could be the sandbox implementation, but there was still an objection. The amendment to require only one implementation was put to a vote, which failed with 10 in favor, 26 opposed.

Further discussion elaborated the procedure to be followed. A proposal would go through the normal process and be provisionally adopted; this means that there is some guaranteed interest in seeing a proposal become part of the standard before implementation work started. In addition, a workgroup could waive the requirement in the event of needing an emergency change for an interoperability problem.

The proposal passed 25-8, though it was noted that this was not the 80% majority required for changes to the actual standard. This generated some resentment on the part of a few of the participants, since the requirements for procedural changes had never been made clear. Bruce Toback agreed to draft a governance document to avoid the situation in the future.

Change Proposals

During the discussion of change proposals, several proposals were determined to be simple clarifications or elaborations of inadequate information in the existing standard. The RETS document editor was instructed to add an "errata" section to the document and the web site that detailed these. Proposals which are to be annotated in the earlier standard are noted below.

RCP040: Media Upload

The media workgroup requested that this proposal be deferred.

RCP041: Padding for Change Password

Passed 24 in favor, none opposed

The group amended the proposal to include relevant standards references: RFC 2616 for Base64 encoding of binary data in the protocol, and RFC 1362 for block padding for DES encryption. Note also PKCS7, an RSA designation that is standardized in RFC 2315.

This proposal is to be added to 1.5 as an erratum.

RCP042: Add capability to request nonstandard DTDs

Rejected with 6 in favor, 14 opposed, with the recommendation that the proposal be taken up by the Metadata workgroup.

Several amendments were proposed for this change proposal, including adding an error code so that an inapplicable DTD could be signaled to a client. In addition, the metadata describing the list of available DTDs should be amended to include a text description of the DTD, and the identifier used in the request should be the public identifier for the DTD, rather than the URL.

However, a number of attendees voiced the concern that certain DTDs would end up being de facto requirements, though nothing in the change proposal requires the support of any additional DTDs. On this basis, and on the basis of the need for additional work, the group voted to send the proposal to the Metadata workgroup for further work.

RCP043: Interpretation of Capability URLs

Adopted, 16 in favor, 4 opposed

Despite the fact that this appeared to be merely a clarification of some ambiguous statements in the specification, it generated considerable controversy because of the potential for incompatibility with existing implementations.

The proposal was amended to make it clear that relative URLs are to be interpreted relative to the address to which the client is connected, rather than relative to the new Login URL. The document editor was instructed to add examples to aid in interpretation.

RCP044: Cookie Handling

Adopted with 27 in favor, none opposed.

The proposal was amended to reference RFC2109 cookies exclusively, and to remove references to the Set-Cookie2 header from RFC 2965.
RCP045: DMQL Changes

Rejected, none in favor, 17 opposed.

After some discussion, it became apparent that there was no sentiment for supporting yet another version of DMQL in the current 1.x RETS architecture. A Search workgroup was proposed to look in to search language issues and produce "one last" language specification.

RCP046: Accept and Content-Type Header Treatment

Rejected with 9 in favor and 8 opposed.

One of the issues in this proposal was the fact that an Accept of text/xml may not be appropriate -- in fact, is not appropriate -- for all requests. The difficulty is in decided whether to send a RETS response or an HTTP 406 response. The suggestion of sending the RETS body with the 406 response was discussed and adopted, but not with much enthusiasm. The consensus was that such a proposal is needed because of the somewhat spurious requirement that the client send Accept�*/*, but this proposal isn't the right one.

RCP047: Action URL Clarification

Passed with 15 in favor, 1 opposed.

The proposal was emended to read that the Action URL MAY be used for a message-of-the-day.

RCP048: Compact Format Simplification

Rejected with 4 in favor and 17 opposed

Apparently a proposal that only its mother could love, this was supported only by the authors. The general consensus was that this proposal represented an unnecessary complication of the standards environment, if not the standard, by introducing a third (or fourth) format that clients and servers would need to support.

This proposal, together with RCP045 and several others, were deferred for study by the workgroup that will author the 2.0 standard, at which point these ideas can be incorporated, if not the actual proposals.

RCP049: Deprecate Multipart GetObject Response

The group postponed consideration on this change proposal, with the intent that it be subsumed into the work product of the Media workgroup.

RCP051: Headers as URL Parameters

Rejected with 6 in favor, 3 opposed

RCP052: General Reply Codes

Passed with major amendments, 23 in favor, none opposed.

This proposal went through quite a number of iterations before finally being adopted. The proposal was initially amended to deprecate rather than delete the transaction-specific miscellaneous failure codes; this was then revoked. The final disposition is that error code 10002 is adopted as specified, and that 10001 is adopted with the proviso that it SHOULD NOT be used if a transaction-specific error code can be determined. In other words, it is designed for failures where the transaction context can't be determined and is expected to be very rare.

There was considerable discussion on the philosophical question of whether or not to use transaction-specific error codes. The argument that carried the day was that the existing transaction-specific codes can be used by a client to generate more meaningful end-user error messages.

Another motion, passed with 23 in favor and no opposition, was passed to require the document editor to clarify the meaning of transaction-specific error messages, and to convey the preference for using them whenever possible.

RCP053: Enhancements to upload transaction

This was punted to the Upload working group for further action.

RCP054: Version Negotiation

Passed with 13 in favor, 0 opposed, with one amendment

This proposal covers the same ground as RCP039, and was also approved with the recommendation that it be added as an erratum to the 1.5 specification. The proposal was amended, by a vote of 16-0, to remove section 4.12, version negotiation.

RCP055: Public Identifier for RETS DTD

Passed with 16 in favor, none opposed

Several participants expressed concern about the need to have a sever "running all the time" that could provide the DTD. The proposal's authors pointed out that the public identifier was just that -- an identifier -- and not necessarily an external reference that must constantly be retrieved. In fact, according to the authors, that's almost never done in practice.

The group recommended that the standard that includes this proposal also include an implementation note explicitly stating that the DTD should be cached locally, rather than retrieved for each instance document that is being read.

RCP056: Change Logon Response Format

Rejected with 2 in favor, 19 opposed.

Again, it was the consensus of the group that this proposal offered too little benefit and caused too many compatibility problems to be worthwhile, especially in light of the fact that a 2.0 architecture would provide many of the same benefits. The proposal will be kept as a reference for the 2.0 workgroup.

Discussion of RETS 2.0 Direction

Bruce Toback began the discussion of RETS 2.0 directions by giving exactly the same presentation he gave exactly one year earlier. The reaction this time, however, was considerably more favorable.

The group approved the formation of a workgroup to create a 2.0 specification using the guidelines outlined in the presentation. The timetable was:

April, 2004 Workgroup presents architectural overview and requirements document to RETS Working Group for comment

August, 2004 Preliminary draft of RETS 2.0 document presented for comment

December, 2004 Final draft of 2.0 document, including reference implementation

RETS 2.0 would become the current version one year later, in December, 2005. There will be a 1.8 document in parallel with the 2.0 document, which will be focused on fixing interoperability problems rather than adding new features. New features will be placed on the 2.0 track.

RETS 2.0 is expected to be SOAP-based. It was recognized that there are architectural issues with SOAP in a high-volume data transfer environment, and one of the principal challenges for the April engineering requirements draft will be a solution to that limitation.