Subject: IPP>PRO: sorry, but binary is better

From: "Sylvan Butler" <[email-masked]>
Date: Thu, 19 Jun 1997 18:41:08 -0700

I'm sorry for the length of this missive (about 150 lines plus
headers...) but it seems necessary.

Last night I revised Paul's document to indicate what we had concluded
on the 17th, and this morning I woke up way too early...  Or perhaps it
was way too late, depending on your perspective.

A binary encoding is MUCH simpler.  Even if limited to just the lengths.
For example, with 16-bit binary lengths (FAIRLY COMPLETE CODE):

// assumes enough incoming buffer to hold entire name and ValLen fields
// requires external help to deal with long or multiple values
int GrabAttr(ATTR *pAttr)
{
  nLength=ntohs((unsigned short)*(U16 *)pBuf);
  pBuf+=2;
  pAttr->nNameLength=nLength;
  pAttr->pName=pBuf;
  pBuf+=nLength;
  nLength=ntohs((unsigned short)*(U16 *)pBuf);
  pBuf+=2;
  pAttr->nValLen=nLength;
  pAttr->pVal=pBuf;
  pBuf+=nLength;
  return ERROR_OK;
}

vs. a n-digit ASCII length (NOT CODE, not even pseudo-c):

// assumes enough incoming buffer to hold entire name and ValLen fields
// requires external help to deal with long or multiple values
int GrabAttr(ATTR *pAttr)
{
  // str functions don't work because we aren't null-terminated
  so we normalize all input buffers?  (null terminate ... ?)
  // or just write our own strtok?
  strtok(ON A COPY of the string)
    // so we can remember the terminating char, either <SP> or <CR>...
  pAttr->pName=pToken;
  pBuf+=strlen(pToken)+1;
  strtok()
  pBuf+=strlen(pToken)+1;
  nLength=private_atoi (because leading 0's would assume octal)
  pAttr->nValLen=nLength;
  pAttr->pVal=pBuf;
  pBuf+=nLength;
  return pBuf;
}
int private_atoi(char *)
{
  do it
}
void normalize() // or private_strtok
{
  do it
}

Both of these examples will require a bunch of strncmp's in order to
actually do anything, because what I've drawn up isn't even as binary
as Paul and I had in the SWP documents (even June 6).  If you reduce
Operation and Attribute names to an enum'd set then all those strcmp's
go away and become a simple '==' or even a switch (YES!).

I think we need to face reality here...  Binary requires significantly
less code to deal with, which means less bugs and less testing, which
means more solid implementations sooner.  (Anybody ever run into a web
site that used atoi and interpreted numbers as octal?  I was just reading
about one last week...).  What do we get with ASCII?  My list of "pros"
from the meeting is really short.  The biggest I've been able to
identify is vendor extensibility.

For extensibility we could reserve the upper 0x8000 (or maybe fewer).
In fact, for attributes we could assign the "all ones" enum and have the
vendor use the first 4+ bytes of the value as their unique attribute
name/ID in order to minimize collisions.

Unless someone has parsing for ASCII all worked out and can illustrate
that it really isn't much more code, I have to be in the binary camp.
IBM's triplets anyone?  (Roger?)

The following three examples show an encoding of a
Print-Job operation with attributes 
job-name=="Spec"
job-originator=="Sylvan", and
  (a multi-value vendor specific hypothetical attribute) 
vendorHWP-BLD-ID=="Alpha",32766

In all examples the bytes on the wire are specified in hex (two
characters) or ASCII (one character).  The spaces between bytes and the
line wrapping are not transmitted.  {print data} is the actual data for
the job and the literal phrase is not transmitted.

-----------
Example 1, ASCII (June17):

 0  1  0  0  P  r  i  n  t  -  J  o  b 0d 0a
 j  o  b  -  n  a  m  e 20  4 20  S  p  e  c  j  o  b  -  o  r  i  g  i  n  a
 t  o  r 20  6 20  S  y  l  v  a  n  H  W  P  -  B  L  D  -  I  D 20  5 20  A
 l  p  h  a 20  5 20  3  2  7  6  6 0d 0a
{print data}

-----------
Example 2, ASCII with binary (version is fixed) lengths:

 0  1  0  0 00 09  P  r  i  n  t  -  J  o  b 00 08
 j  o  b  -  n  a  m  e 00 04  S  p  e  c 00 0e  j  o  b  -  o  r  i  g  i  n
 a  t  o  r 00 06  S  y  l  v  a  n 00 0a  H  W  P  -  B  L  D  -  I  D 00 05
 A  l  p  h  a 00 00 00 05  3  2  7  6  6
{print data}

-----------
Example 3, binary (SWP, IBM's triplets...)
(extra spaces, line wraps, and comments added for clarity):

00 01 00 00                            ; version 01.00
00 01                                  ; Print-Job
00 01                                  ; job-name
00 04  S  p  e  c                      ; length and value
00 02                                  ; job-originator
00 06  S  y  l  v  a  n                ; length and value
ff ff                                  ; vendor--see 4+ bytes of value
00 0a  H  W  P  -  B  L  D  -  I  D    ; length and value1
00 00                                  ; additional value for prev attribute
00 05  A  l  p  h  a                   ; length and value2
00 00                                  ; additional value
00 02 7f fe                            ; length and value3
{print data}

-----------

Going from example1 to example2 eliminates scanning and tokenizing.

Going from example2 to example3 eliminates strcmp's to match names
of attributes and operations.

I vote to go all binary, eg. #3.  (Note though, that I encoded the HP
attribute name in ASCII whereas in real life I'd probably pick more like:
  H W P xx yy zz
where hex yyzz would increment but the rest would remain static for all
the attributes that I create.  I probably wouldn't do the multi-valued
attribute either, rather I'd just append onto the first value since the
prefix length is fixed.)

I apologize for having to do this, but evidently my implementor hat
wasn't fitting very well on June 17th.

Thanks for your attention,

sdb

 | Sylvan Butler | [email-masked] | AreaCode 208 Phone/TelNet 396-2282 |

---

From: "Sylvan Butler" <[email-masked]>
Date: Fri, 20 Jun 1997 09:49:24 -0700

Bob Herriot wrote:

>binary protocol. Also, my ANSI C book states that atoi assumes decimal, but
>recommends using strtol as I have done below. 

Ahh, yes.  How did I miss that...

>  char * pNext; 
>  int length; 
>  pNext = strchr(pBuf,' '); 

Probably safe given my initial assumption that the buffer would hold 
the entire name...  But hard for the upper layer to validate.

>  pAttr->nNameLength = pNext-pBuf;
>  pAttr->pName = pBuf
>  pBuf = pNext + 1;
>   
>  length = strtol(pBuf,&pNext,10);  /* ANSI C function */

Doh!  Hardcoding the base.  I missed that one.

Again, we have the potentially hard to verify issue of searching 
forward.

>  pAttr->nValLen = length; 
>  pNext++; 
>  pAttr->pVal = pNext;
>  pBuf = pNext + length;
>  return ERROR_OK 

Looks good, better than I expected.

I am concerned with how the upper layer can validate the assumption 
that the delimiters will be in the buffer for this lower layer.  A 
"pure" text (with max line lengths defined, CRLF endings, etc) can 
read until it gets the entire line (or declare an error when the 
line is found to be too long) and know it has the whole thing.  With 
unknown line lengths it is risky to assume that without verification.

With the binary lengths the GrabAttr doesn't loop searching for 
anything and the layer above can compare lengths with the len of data 
in the buffer to verify that all is well.

>would expect than an implementation might have a keywordToInt function
>which would map keywords to integers that in turn could be used in

Of course, but remind me again, what does an implementation gain by 
having such a function as opposed to just communicating in binary?

>> From [email-masked] Thu Jun 19 18:06:36 1997
 ...
>> What do we get with ASCII?  My list of "pros"

[proposal, examples, etc. deleted]

sdb

---

From: "Sylvan Butler" <[email-masked]>
Date: Fri, 20 Jun 1997 09:50:40 -0700

...

sdb

---

From: [email-masked] (Robert Herriot)
Date: Fri, 20 Jun 1997 14:38:54 -0700

You raise an important issue about a buffer that may contain only a
partial parameter (attribute) but I come to a very different conclusion
as I explain below.  I think that the 'name ":" value CRLF' may again
be the simplest solution from an implementation stand point, even
though we keep rejecting it.  The reason lies with the way the buffers
align with parameters (attributes).

This buffer problem exists for both the current IPP proposal and for
the previous binary proposal that you prefer.  The two byte binary
integer could span buffers and a parameter name or value buffer could
span buffers. Similar spans could occur with the current IPP protocol
proposal. The code to deal with processing an attribute that spans two
buffers seems rather messy to me because the break can occur during any
element. It sees like a good place to generate lots of occasional bugs.

You observed that with the two-byte lengths in the binary proposal,
it is easy to determine if the buffer ends in the middle of the string.
It is also easy in the current proposal to get the same information by
putting a " " (space character) just after the last character in the
buffer to stop the scan-for-space algorithm.

You also observed that with CRLF and a maximum line length, it is easy 
to ensure that a buffer has a full unit because functions, such as fgets
read upto the next CRLF. Perhaps we need to revisit the solution that
uses CRLF, namely the one where each parameter has the syntax

   name ":" value CRLF .

The syntax for such a solution would be

    parameter = single-value / set-of / continued-value
    single-value = parameter-name ":" value CRLF
    set-of = parameter-name 2*( ":" value CRLF)
    continued-value = " " value CRLF

from a raw level the syntax for a parameter is:

    parameter = 1*998 BYTE CRLF
    BYTE = %x00..%x09 / %x0B / %x0C / %x0E..%xFF  ; any value except CR or LF

    CR in a value is represented by =0C
    LF in a value is represented by =0A
    =  in a value is represented by =3D

The following is the GrabAttr function is designed along the lines of your
function except that it returns the type of the parameter (attribute).
It assumes that the buffer holds at least 1000 bytes and ends with a
"CRLF" so there are no partial parameter issues to deal with in the
GrabATTR function. I am assuming that the caller of GrabAttr copies the
strings referenced by pAttr and for the value it translates an =xx to
the actual character.  I am also assuming that it used the value
returned (attrType) to determine how to use the attribute just read.

  int attrType;
 
  switch (*pBuf) { 
     case ' ':  /* contination line /*
        attrType = CONTINUED_VALUE;
        pBuf++;
        break; 
     case ':':  / another member of a set */ 
        attrType = SET_OF;
        pBuf++;
        break; 
     default: 
        attrType = SINGLE_VALUE;
        pNext = strchr(pBuf,':');
        pAttr->nNameLength = pNext-pBuf;  /* this works now */
        pAttr->pName = pBuf;
        pBuf = pNext + 1;
     }   
  pNext = strchr(pBuf,'\r');
  pAttr->nValLen = pNext-pBuf;  /* this works now */
  pAttr->pVal = pBuf;
  pBuf = pNext + 2;             /* skip CRLF */
  return attrType;
}

Perhaps this pure textual solution is easier to program once you consider
the buffer problem.

Comments?

Bob Herriot

> From [email-masked] Fri Jun 20 08:51:08 1997
> 
> Bob Herriot wrote:
>

---

From: "Sylvan Butler" <[email-masked]>
Date: Mon, 23 Jun 1997 10:02:48 -0700

>integer could span buffers and a parameter name or value buffer could

The upper layer can trivially ensure that the buffer contains X bytes 
of data, where X is large enough to hold at least a two byte attribute 
name and a two byte value length.  Once the lengths are known then 
the proper amount for values is easy to collect.

With an ASCII encoding the upper layer has to guess, or check for 
proper terminators.

>element. It sees like a good place to generate lots of occasional bugs.

Yes, except with binary it is not messy at all.

>buffer to stop the scan-for-space algorithm.

That would work, if you must scan.

>read upto the next CRLF. Perhaps we need to revisit the solution that

Unless you have to interoperate with implementations that visually 
checked (instead of looking at a sniffer hex dump) for "lines".  Like 
with HTTP servers today, even though the spec reads CRLF you still 
find CR (from mac?), LF (from unix?) and the desired CRLF (I wouldn't 
be suprised to find some LFCR's out there).

>    =  in a value is represented by =3D

You will probably need to escape anything less than %x20 and perhaps 
anything greater than %x7E.

In fact, instead of developing our own escape rules we should, as you 
probably did, just pick one.

This is just so ugly, considering the purpose is for two computers to 
talk to each other over a link that is 8-bit clean.

>Perhaps this pure textual solution is easier to program once you consider

I don't find it so.

sdb

---

From: [email-masked] (Ira Mcdonald x10962)
Date: Mon, 23 Jun 1997 16:50:29 PDT

Hi Stuart,

I'm just a guest here, but I agree with you.  The 'code fragment wars'
are all very amusing, but miss the point of reliability of implementations
entirely.  The current state of HTTP stuff with 'CRLF' just reinforces
this view.

If you use an 8-bit clean channel and then coerce your attribute
names (identifiers) and lengths to 7-bit 'printable' ASCII, all you
gain is that the long-term hope of good quality localization support
for other languages and other code sets is lost irretrievably.  It
simply won't get realized, because it's too much hassle for the
implementors.

Justifying the protocol attribute encoding because the first
transport is HTTP is nearsighted and an argument that the IETF
area directors certainly won't sympathize with.

Cheers,
- Ira McDonald (outside consultant at Xerox)
  High North Inc
  PO Box 221
  Grand Marais, MI  49839
  906-494-2434