The December 2002 RETS meeting was attended by 41 people representing some 30 organizations. About two-thirds were new to the meeting process. Dale Stinton of NAR made some introductory remarks, and we reviewed the working group group procedures and history for those new to the process.

II. RETS 1.5 Discussion

The nominal release schedule calls for the RETS working group to accept change proposals for discussion at each semiannual meeting. Those proposals approved at the semiannual meeting are then incorporated into the standard, published, and implemented by the early adopters. The standard is approved with changes at the following semiannual meeting.

Comments and Changes

There were a number of comments on the current draft of the RETS 1.5 specification.
1. Paul Stusiak questioned the specification's use of TCP port 443 for a RETS server using SSL. He noted that the specification recommends the use of IANA-registered port 6103 for unencrypted RETS, and should also have a registered non-privileged port for RETS over SSL. Several people dissented, saying that this makes firewall configuration difficult. However, a poll showed that about a quarter of the participants use the IANA-registered port rather than port 80. In addition, others also seconded the concern about having a non-privileged yet standard port available. We agreed to request a second port from IANA to handle RETS-over-SSL.
2. Paul Hessman noted several outdated RFC references in the existing document. These will be corrected in the second draft. In addition, he noted that there is a standard Backus-Naur Form (BNF) for RFCs, and he recommended adopting that. If the changes are minor, this will also be incorporated in the next draft. Finally, Paul pointed out that the document should use UTC in place of GMT. This will also be done.
3. The standard has no definition of a "foreign key ID", a term used in Section 11. This will be added.
4. Several people requested an index of compliance items. This will be added to the next draft.
5. Mark Lesswing noted that the draft calls for URIs to be transmitted at logon, but that some people were using URLs. The specification should be changed to indicate that URLs rather than URIs should be passed back in the capability list.
6. It was noted that the DTD draft issued with the specification was not well-formed, and the wrong working copy had been issued as the draft. The correct draft needs to be substituted before final acceptance. Noted by Paul Stusiak
Clarifications

In addition, there were several requests for clarifications in the specification. Brita Brodin noted two issues: the specification should make clear that METADATA-SYSTEM is the root of the metadata tree (this is mentioned in the specification, but apparently not loudly enough), and that foreign keys shouldn't substitute for denormalization of data records. Stuart Schuessler, who authored the foreign key change proposal, concurred with this request.

Several people were unsure how to implement the new History well-known resource type. Leo Bijnagte, who authored that portion of the DTD, clarified its use, particularly with respect to the <property> tag. His clarifications will be added to the specification, and an implementation note will be written to help new developers.

Approval

While the group protocol calls for a straight up-or-down vote on the current draft at the meeting, it was decided that the draft should be approved "with the discussed changes." Thus, the next draft, after a short comment period for textual corrections, will be the final 1.5 standard.

III. Compliance and Interoperability Discussion

Since the topic of interoperability and compliance was recently a popular topic on the rets-dev list server, a session on these topics was added to the meeting agenda. The discussion resulted in several recommendations for new sections in the specification, user education and testing tools:
1. Many people liked the idea of a "compliance rating": something like "Gold, Silver, Bronze" or "Diamond, Emerald, Quartz". However, there was little agreement on how to select ratings. There was general agreement that compliance should not be based on a count of checkboxes of "optional" features: some products may simply not need to implement the optional features, particularly on the client side of the wire. A server that's intended only for mirroring or IDX/VOW use might well not support update, yet be completely compliant in every other respect. More work is needed, perhaps to define compliance categories orthogonal to compliance ratings. At minimum, there should be different compliance matrices for clients and servers, and any checkbox scheme should include a "doesn't need/not applicable" checkbox.
2. One person suggested the idea of a "compatibility matrix", hosted on the RETS site or on a user-oriented RETS site. This would show which clients work with which servers. The idea was developed further so that perhaps each intersection in the matrix could be a link to comments made by users who are working with that particular client/server pairing.
3. The login transaction can't be optional. It's not clear that it is in the specification.
4. Standard Names were a major concern. Some mid-sized vendors don't implement them at all, yet without them, much of the usefulness of the specification is negated. There needs to be a "site compliance checklist" as well as a software compliance checklist. This is likely to be an education issue: since it's the MLS operators who need to come up with StandardName mappings, it needs to be clear that it's important to do so. Some people suggested that any rating scheme should include a category for "number of standard names", but it was pointed out that this was really beyond control of the vendor and is a site compliance item. The consensus was that there should be a policy statement and a concerted education effort to increase the number of StandardName mappings in data bases.
5. Since the specification has quite a number of optional items, one person suggested the inclusion of a RETS transaction that can tell a client what capabilities are implemented in the server.
6. It was pointed out that some of the optional compliance items that are effectively site compliance issues may in fact be against an association's business rules. Again, there needs to be an education effort for these issues, and compliance measurement scheme needs to take this into account.

A compliance workgroup was formed to look into these issues (see below for more on the workgroups).

An additional suggestion, made later in the meeting, was the idea of a "RETS Plugfest": a developer-level meeting at which participants would actively connect clients and servers to see where the problems were. The proposed meeting would last a day, and would be held at a facility with a high-speed Internet connection so that server vendors wouldn't have to bring machines to the meeting. The room would be equipped with whiteboards, a protocol analyzer or two, and perhaps some other equipment to facilitate engineering collaboration to iron out any problems. Dale Stinton of NAR said that he would look into having NAR sponsor such a gathering, since the improved compatibility would be very valuable to NAR's membership.

IV. Change Proposals

There were four change proposals submitted in time for publication in the meeting package. In addition, however, one vendor submitted a package of eleven change proposals the night before the start of the meeting. This made it clear to the group that a policy was needed to prevent last-minute proposals. The group decided that change proposals must be submitted at least four weeks in advance of a meeting to be considered at that meeting. In the event, two of the vendor's proposals were considered, as noted below.

RCP014: Metadata Backward Compatibility

This proposal generated a large amount of discussion in the meeting as well as on the developer mailing list. The crux of the discussion was how much effort a server should put into backward-compatible metadata. Several alternative schemes were offered to accomplish the same thing, and it was generally agreed that more work would be needed to solve the problem; this proposal doesn't solve the problem effectively in some situations. The alternatives/augmentations included:

• Queryable metadata "change notes" to allow a client to see what changed. One opinion was that this was the perfect solution; another was that the client should do the comparison so that it can determine which fields of interest to it had changed, and a third was that the second was insufficient because the client couldn't detect some kinds of semantic changes.
• In response to the last of these three concerns, someone offered the idea of an unchanging "ID" field on each metadata line item, which would allow a client to know about substitutions of one metadata item for another. The discussion of this idea included the notion that sometimes, it would be meaningless to supply such an ID because the server couldn't know what the client would consider to be a "change in place".
• A third idea was to add a Min-Metadata-Version tag in more places in the metadata so that the client could know when it would become impossible to use the metadata it currently had on hand.

RCP015: Ordering of RETS Responses

Sentiment on this proposal was almost cleanly divided along client/server lines: most client vendors liked it; no server vendors did. The main concern was the potential uncontrolled performance impact of a client requesting a sort of a many-thousand-record retrieval. While the proposal includes several features for controlling the impact of implementing this feature, participants expressed the concern that a server that took advantage of these mechanisms (marking only indexed fields sortable, or using the "sort too complex" reply code) would be viewed as deficient in implementing the specification.

On the client side, there were two arguments. The first was that very thin clients may not have enough storage capacity to handle a sort, and the second -- more widely viewed as important -- is that the RETS Search transaction includes an Offset field, allowing the client to retrieve data in chunks. This isn't meaningful unless the order of the data can be guaranteed.

Several amendments were offered to either solve the problems with the proposal or address the concerns that the proposal was written to deal with:

• Amend the proposal to state that only indexed fields need to be marked as sortable. This was rejected because it was deemed insufficient either to ameliorate the performance concern (since a multi-level sort specification would still require the server to sort), or to solve the perceived-as-deficient concern outlined above.
• Amend the specification so that a client can indicate that it is doing chunked retrievals. This could allow a server to issue records in its natural order, insuring that a query using the Offset parameter would always be indexing into the same list. This was considered harmless enough, but would not completely solve the concern since the search result could still change between transactions.
• Allow a server to indicate which presorted values are available. Nobody liked this either.

RCP017: Add a RETS Namespace

This proposal adds a RETS namespace to the standard DTD. The proposal was made both to facilitate adding Association-specific extensions and to improve interoperability with other standards. Several deficiencies were noted in the proposal, and it was adopted with the following changes:

• The namespace identifier is to follow W3C guidelines, and to have the form of a URI.
• The proposal will contain usage guidlines for mixing RETS and local tags.
• The proposal will be amended to state that the RETS namespace is the default when a search requests a STANDARD-XML format for its return.

This change proposal was part of the package in the late submission. However, it was adopted because it is merely a textual clarification. This engendered a discussion on the need for usage clarifications elsewhere in the metadata, and it was agreed that this would be done generally. The metadata description often describes only the form and not the anticipated usage for metadata fields.

RCP020: Allow .ALL. As Search Value

Initially, this proposal, which allows a client to specify some token such as .ALL. to indicate that it wishes to find all values of a lookup field, was fairly well-supported. However, several participants noted implementation concerns:

• How would the .ANY. token interact with the MaxSelect field in the metadata? Would it override MaxSelect? Would the server be able to choose an arbitrary number or set of lookup values?
• The stated reason for this change was that some servers have lookup items designated as mandatory for searches. This means that if the user wants to search all records for something other than the lookup item, the client needs to specify all of the potential values, and in some cases, there may be hundreds. It was noted, however, that the reason for making a lookup item mandatory may be precisely to limit the scope of searches to something that the server can handle, and including this capability would break some servers.
• Should .ALL. include null values? The proposal doesn't say.

V. Additional RETS Resources

Steve Verba of Avantia announced that his company had released an NAR-sponsored reference implementation of RETS 1.5, as well as an open-source RETS client in Visual Basic. He also stated that Avantia is working on a scriptable, open-source compliance tester, and provided an early version so that developers could examine it and possibly begin writing scripts.

VI. RETS 2.0

Bruce Toback began the discussion with a short presentation on the history of RETS and a proposed set of goals for RETS 2.0. This was intended to seed the discussion and begin a collection of objectives and basic choices for RETS 2.0. The presentation suggested the use of a two-track approach, continuing to enhance and support RETS 1.x while working on a 2.x release for an 18-24 month timeframe.

Surprisingly (to Bruce, anyway), there was overwhelming, though not unanimous, sentiment for not starting on a specific RETS 2.0 proposal. Though the RETS group would eventually release a 2.0, we should reach that goal by incremental change using the existing change proposal mechanism. The concern was that some developers, knowing that a 2.0 was coming and that it might be substantially different, would delay development work until 2.0 was published. The fact that the 2.0 process would be open and that interim drafts would be public was not considered adequate to overcome this concern.

With that out of the way, the next order of business was to formulate specific needs for moving the standard forward, and to organize the work to create change proposals to address those needs. Participants came up with five major areas for new work:
1. A more comprehensive data dictionary. Many participants stated that the list of standard names was inadequate and that more work should be done on it. A data dictionary workgroup was formed to enhance the RETS 1.5 data dictionary and begin adding more elements. There was very little sentiment for moving to XML Schema.
2. Separate the RETS payload from the transport. The idea was to be able to adopt a richer data dictionary or a more standard transport mechanism independently. The richer data dictionary should certainly be available to RETS 1.5. Opinion was sharply divided on the issue of transport. One faction thought that SOAP and web services were unlikely to survive, while the other faction stated that RETS would not survive if it did not adopt new transport. Clearly, there is scope for discussion, and a transport working group was formed to work on the issue.
3. Allow publishing of procedures. This is closely linked to transport, since adoption of a standard procedure-publishing mechanism such as Web Services Description Language (WSDL) implies adoption of a standard transport to go with it. The transport working group will look at this issue as well.
4. Add data dictionary chunks from other standards. This will be looked at by the Data Dictionary working group. It was recommended that RETS fold in relevant chunks of property description namespaces from other groups, notably MISMO, and specify when and how those extensions should be used.
5. Revisit the Update transaction. A separate working group was also formed to look at this. In particular, the needs are for a stronger validation mechanism as well as an XML upload capability.

The group strongly favored continuing work on 1.0, incorporating the long-range work wherever possible. This was the most popular path to get to a 2.0 standard. In addition, a use case document should be created for RETS, to guide the enhancement efforts as well as an aid to adoption.

VII. Next Meeting

The group adopted a suggestion to change the semiannual meetings from June/December to February/August. The next meeting will be scheduled in late February, where it will follow the RETS Plugfest. The working groups are to have their proposals ready well in advance of that meeting.

The working groups will meet by conference call in order to facilitate the drafting of revised change proposals. This will begin in the next two or three weeks. Details on joining the conference call will be posted on the web site so that participants who were not at the face-to-face meeting can still help shape the proposals. (In at least one case, the proposal author had a last-minute emergency that prevented his attending.)

-- Bruce Toback