Subject: SENSE> SENSE Req review

From: Harry Lewis <[email-masked]>
Date: Wed, 5 Nov 1997 18:20:00 -0500

I have reviewed the SENSE requirements doc (v1.0 11/28/95) as requested
 and have the following comments:

1. (Overview)  In general, while current NMS and Printer MIB solutions seem
to be performing adequately, I agree with the underlying premise of SENSE,
that there is room for widespread application of event driven network device
management and view this as an evolutionary step.

2. (Overview)  Simple is stressed as an overriding characteristic of the
requirements. I wonder if the current model, with it's publication and edition
paradigm is really as simple as originally intended. This would be unfair as a
criticism (and is not intended as such) since I do not have an alternative
model to offer. It's just an observation that the current state of the
architecture may not fit the icon of simplicity put forth in the requirements.
Simple is a very hard thing to define and it may be better to avoid the term.
Given that events really capture the essence of the situation perhaps the name
should be changed to ES&NSE (Event Subscription & Notification Services
Environment).

3. (SENSE Requirements)  The primary motive... lack of good Trap registration
and distribution in SNMP may well be handled in SNMPv3. This motivation should
be re-investigated.

4. (#14)  Client ACK of event. I'm not sure this should be required in all
cases. Perhaps 2 modes, an ACK or NotACK mode. This will help alleviate (#16) -
server resource management. The server can really bloat while waiting for ACKs
from clients who disappear etc. It is feasible to consider an "unreliable" mode
of operation.

5. (not addressed)  The requirements do not appear to directly address the need
for filtering. I know the requirements do not address data content and it could
be argued this is where this topic belongs, but I think filters are significant
to any event based scheme and should at least be mentioned. I also acknowledge
that filtering is exactly one of the reasons the current SENSE architecture may
appear somewhat less than simple (my comment 2, above).

Harry Lewis - IBM Printing Systems

---

From: Jay Martin <[email-masked]>
Date: Thu, 06 Nov 1997 13:50:04 -0500

Harry,

Thanks for your comments.  I wish others would take the time and
do the same, or at least post a message to the list saying they
are in full agreement with the document's contents.

> should be changed to ES&NSE (Event Subscription & Notification Services

I can certainly understand how you might think the Publication/Edition model
is not very simple.  I think the reason for this is that we have not yet
fully discussed the requirements as mapped to a proposed design/implementation.
Perhaps once we start looking at potential designs (including the existing
design as conceived by Underscore), then folks will be able to better judge
whether the Publication/Edition model is useful, or whether something else
should be pursued.

For the time being, though, let's leave the project name as it is (ie, Sense).

> and distribution in SNMP may well be handled in SNMPv3. This motivation should

Yes, you and I have privately spoken about this situation.  Sense, having pre-
dated SNMPv3 by quite some time, may now need to be reexamined to see if
SNMPv3 eclipses the need for Sense, or at least certain parts of it.

I need to do another head-to-head competitive analysis of SNMPv3 against Sense,
similar to what I had previously done for DMI v2.0.  Hopefully I'll be able
to get to that in the next week or so.  (By the way, can anyone give me some
quick net pointers to the SNMPv3 docs?)

> from clients who disappear etc. It is feasible to consider an "unreliable" mode

Excellent observation, Harry.  We, too, have found that an "unreliable mode"
is certainly viable in certain environments.  This shouldn't be too hard to
add to Sense.  Do others have any comments on this addition?

Right off the top of my head, I'd say that the subscription request would be
defined to have one or more message properties indicating the level of reliability
for events delivered for that subscription.

The client API in the current implementation actually supports the kinds of
reliability controls you mention.  However, these controls are used across the
client session and can not be altered on a per-subscription basis.  Perhaps such
session-wide controls are enough for you, though.

> appear somewhat less than simple (my comment 2, above).

Ah yes, filters.  As I described in Boulder last week, filters, or more precisely,
filter specifications, are one of the primary reasons we conceived of the notion
of an "Edition".  Filter specifications can get very, VERY ugly, and before you
know it, you often end up designing an entire "language" to deal with that.

This is something we at Underscore desparately wanted to avoid.  I'll address
that issue in more detail in a later message.  However, for now, suffice to
say that a requirement exists for a client to be able to specify receipt of only
a subset of all events that may be available from a given source.  Exactly how
that "subset" is defined and handled is a topic for design and implementation,
IMHO.

	...jay

----------------------------------------------------------------------
--  JK Martin               |  Email:   [email-masked]          --
--  Underscore, Inc.        |  Voice:   (603) 889-7000              --
--  41C Sagamore Park Road  |  Fax:     (603) 889-2699              --
--  Hudson, NH 03051-4915   |  Web:     http://www.underscore.com   --
----------------------------------------------------------------------

---

From: Tom Hastings <[email-masked]>
Date: Fri, 14 Nov 1997 12:11:36 PST

>session-wide controls are enough for you, though.

I agree that the subscriber should be able to indicate whether he/she
want reliable or unreliable.  (It shouldn't be a propertly that only
the publisher sets.)

I thought that there was a property in the edition that specified the 
number of re-tries before giving up.  Since there must be a separate 
counter for each subscriber to keep track of how many retries have been
made, couldn't the subscriber also specify the number of retries when
subscribing?

A value of 1 would mean unrealiable.

>that "subset" is defined and handled is a topic for design and
implementation,

Yes, allow the subscriber to specify a simple subset of the 6 defined
events (actually only 4, because 'all' and 'none' are really separate
events).

In IPP, we defined the following set of events and allowed the client
to specify any combination of them.  That doesn't sound too difficult.

Part of a publication should be a single property which is the names of
the set of possible events.  The subscriber specifies a property which is a
subset of that publication property.

Here is the following text from the 10/14/97 Model draft (that was
subsequently deleted because of a lack of a transport for the events).

4.2.2	notify-events (1setOf type2 keyword) 

This attribute specifies the events for which the end user desires some
sort of notification.  The "notify-addresses" attribute is used to describe
the destination addresses for these events.

Standard values are:

'none': the Printer SHALL not notify. 

'all': the Printer SHALL notify when any of the events occur.

'job-completion':  the Printer SHALL notify when the job containing this
value completes (i.e., enters the 'completed', 'canceled', or 'aborted'
state) with or without errors.

'job-problems':  the Printer SHALL notify when this job has a problem
(i.e., when the job leaves the 'processing' state and enters the
'processing-stopped' state).

'job-started-processing':  the Printer SHALL notify when the Printer starts
processing the Job (i.e., when the job leaves the 'pending' state and
enters the 'processing' state).

'printer-problems': the Printer SHALL notify when this job is affected by a
Printer problem.  This happens when the printer enters the 'stopped' state
while this job is in the 'pending', 'pending-held', 'processing', or
'processing-stopped' state. 

In a create request, if the client supplies 'none' along with any other
combination of values, it is the same as if only that other set of values
had been supplied (that is 'none' has no affect).  If the client supplies
'all' along with any other combination of values, it is the same as if only
'all' had been supplied (that is 'all' subsumes all other values).

---

From: Jay Martin <[email-masked]>
Date: Sat, 15 Nov 1997 17:01:56 -0500

> want reliable or unreliable.  (It shouldn't be a propertly that only

I agree 100%.  No issue here.

> made, couldn't the subscriber also specify the number of retries when

No, there is no "retry" property in an Edition.  Retries are handled
at the client level; that is, during client registration and potentially
at per-Edition subscription time.

In other words, I believe Sense is already designed to handle what
you'd like to see, and if there are any holes remaining, we can
certainly fill them using the kind of approach you and Harry have
suggested.  No issue here.

> A value of 1 would mean unrealiable.

Actually, I'd suggest a value of 0 to indicate "no retries" (ie, a
retry count of zero).  Seems to be more elegant and useful, no?

To make this message a bit more readable, I have included the rest
of your message (with Harry's and my embedded comments) at the end
of this message.  Following are my comments to that text.

First, I think we're pretty much in agreement with the basic Edition-
oriented philosophy that precludes the need for extensive filter
specification, etc.  (I realize Harry may not agree with this.)

As you mention, the prior established specifications for IPP job-
related events reception by a client are:

  none
  all
  job-completion
  job-problems
  job-started-processing
  printer-problems

Of course, in the case of Sense, the "none" case is not relevant;
the Subscriber client simply does not subscribe to any of the
associated Editions.

Separate Editions can be designed for the remaining types, though.
To keep life simple, it would be best for us to focus on the "all"
case, since I honestly believe this specification will be used by
most clients.  (Agreed?)

What we need to do then is:

  1) Name the Edition.  Something like "IPP.Job.AllEvents" would
     follow what we have done in the Printer MIB area, but this
     choice is obviously quite arbitrary.  What can be important,
     though, is the left-to-right dotted-string convention for
     naming things in a rough hierarchical manner.

  2) Define the Event message format.  A standard Edition property
     (Edition.Protocol) is used for this purpose.  Again, there is
     tremendous leeway in defining the message format.  At Underscore
     we spent quite a bit of time trying to come up with a generic
     event format mechanism that can work for many, many different
     kinds of event scenarios.  We call this the "Simple Text Event
     Protocol, Type 1" (STEP.1), and documentation exists on the PWG
     Sense area describing this format and how it's used.

     We can take a crack at defining IPP Job messages using STEP.1,
     or devise and entirely different method.  Makes no difference
     to us.  One of the beauties of Sense is that it is defined to
     allow the existence of parallel Editions, where the same content
     and semantics is provided, but the format differs (for whatever
     programmatic reasons that might exist).

  3) Defined the exact contents of the message (independent of the
     wire protocol, or "format" as I have referred previously).  Here
     we have a pretty simple task, since we should merely leverage
     the data and format as (previously) defined in the IPP Model
     document.

What I'd like to instill at this point is that defining an "Edition"
(or multiple Editions) for IPP Job events is pretty trivial...assuming
that you buy into the Sense architecture as currently defined.

We must first get the architecture right before spending time on
more concrete applications of Sense.  Hopefully folks can see how
easy it would be to define all kinds of Sense Editions for IPP-related
events, but first we need to all agree on the actual architecture.

Agreeing on the Sense architecture is what I hope to accomplish at
the upcoming LA meeting next month.  Assuming, of course, that more
input/comments/etc are received on the Sense DL before the end of
next week.

	...jay

PS: To help folks visualize things a bit bettter, here is the full
    contents of an Edition providing all device events for a printer
    based on the Printer MIB:

Edition.ClientId = 6
Edition.Content = AllDeviceEvents
Edition.Protocol = STEP.1
Edition.Publication = printer/digital/lps/17|dorky|underscore.com
Id.Class = app/underscore/pa/publisher/sfplps/editions/step.1/AllDeviceEvents
Id.Client = 6
Id.Contact = 
Id.Description = All printer device events
Id.Domain = underscore.com
Id.Host.Name = uscore
Id.Id = 10
Id.Manufacturer = Underscore, Inc.
Id.Name = dorky
Id.Product = PrintAlert
Id.SenseVersion = 1.0
Id.Signature = 
Id.Updated = 878572337
Id.URL = http://www.underscore.com/products/pa/publishers/sfplps.html
Id.Version = $Id: .edition 1.1 1997/01/17 09:52:02 jkm Exp $

Additional comments from Tom/Harry on message filters and IPP job

---

From: Tom Hastings <[email-masked]>
Date: Tue, 18 Nov 1997 11:08:28 PST

>Tom Hastings wrote:
snip...

>retry count of zero).  Seems to be more elegant and useful, no?

I agree. Its a retry count, so 0 means no retries.

snip...

>     naming things in a rough hierarchical manner.

While "all" might be the most used, it depends on the operator coverage
of the printer and whether the user likes lots of notifications or not.

For printers that have operator coverage, only getting "job-completion"
would be most used.

I also found the "job-started-processing" event annoying, though some
like it a lot.  It depends on whether the print system responds in the
create job response with whether the job was started or is pending.
If the print system does (and VMS did), and for printers that are often 
idle (10% duty cycle), just knowing that the printer was idle when the 
job was submitted is better than being notified asynchronously (usually
almost immediately) that the job has just started.

In other words, I belive that which of the above combinations of events
is very much a user preference issue.  Does SENSE have any problems
with allowing users to choose any combination?  Does allowing users to
choose mean that there has to be combinations of editions for each
combination, each with a different name?  Is this a problem to have
to have so many combinations?

If yes, then I'd like to see if we can fix SENSE architecture
so that the SENSE server does a minimal amount of filtering as stored
with the subscription.  The publication lists what are the (short) list
of event groups and the subscription says which event groups are wanted.
The SENSE server ignores sending notifications to subscribers that
do not have the indicated event group in their subscription.

Would this be feasible?

Tom

snip...

---

From: Jay Martin <[email-masked]>
Date: Fri, 21 Nov 1997 11:47:39 -0500

Tom,

> do not have the indicated event group in their subscription.

As I have stated many times, I am very much against the specification
and implementation of event filter specifications in Sense, due to
the fact that such filter specifications typically turn into a full-
blown language...with all the attendant nightmares that go with it.

We tried to strike a balance between simplicity and flexibility in
providing sets of events via the concept of "Editions".  The producer
(ie, Publisher) defines event sets and publishes Editions to represent
those event sets.  The client then subscribes to the event sets of
interest; when an event arrives for a given subscribed Edition, the
client decides whether the event is "of interest" (ie, filters the
event) and handles it accordingly.

I am very much against the notion of having the protocol/system design
provide Nth-degree filtering.  While such filtering can definitely
improve network performance, it comes at a considerable system design
and implementation cost.

We explicitly took the path of "reasonable compromise" in that event
sets are segregated (using Editions), but the client performs local
filtering as desired by the user.

	...jay

Subject: Re: SENSE> SENSE Req review
From: Tom Hastings <[email-masked]>
To: Jay Martin <[email-masked]>
Cc: Harry Lewis <[email-masked]>, [email-masked]
Date: Tue, 18 Nov 1997 11:08:28 PST

snip...

snip...

Would this be feasible?

Tom

snip...

---

From: Tom Hastings <[email-masked]>
Date: Fri, 21 Nov 1997 10:04:47 PST

>event) and handles it accordingly.

What we keep trying to tell you is not to be afraid of simple filters.
I agree with you that filters can get out of hand.  But for IPP 
for end-users monitoring their jobs and/or the printers to which they
have sent jobs, the filter mechanism is just four bits.

Please don't ignore the "simple" IPP requirements because of a fear
that the filter mechanism will get out of hand.  Lets just design a
simple filter that can be expressed as a single small set of bits.
Its not a full fledged language.  There is no BOOLEAN expression
to evaluate.  Its just whether the event being generated is one of the
event groups that the subscriber is asking for.

In making a general SENSE mechanism, limit the filter to 32-bits,
which is checked in a single AND operation to see whether the event
being generated is among the requested event groups or not.

I repeat the simple IPP requirements (which seem to keep getting deleted):

Allow the subscriber to specify a simple subset of the 6 defined IPP
event groups (actually only 4, because 'all' and 'none' are really NOT
separate
events).

In IPP, we defined the following set of event classes and allowed the client
to specify any combination of them.  That doesn't sound too difficult.

Part of a publication should be a single property which is the names of
the set of possible event classes, with a limit of 32 classes.  
The subscriber specifies a property which is a subset of the 
event classes of that publication property.

Standard values are:

>improve network performance, it comes at a considerable system design

Do you consider my suggestion above too complicated?

Do you think that it is insufficient?

>filtering as desired by the user.

But in order to meet the IPP requirements of any combination of
4 event classes, the publisher would have to create 16 editions, correct?

If 16 editions isn't a problem for each publication, then we don't need 
the simple filtering that I'm proposing, but I had assumed that each IPP
printer having to set up one publication with 16 editions for job submission
was too much.

Am I wrong?

Tom

>----------------------------------------------------------------------Recei
ved: from alpha.xerox.com (alpha.Xerox.COM [13.1.64.93]) by
uscore.underscore.com (8.8.4/8.7.2) with SMTP id RAA10158 for
<[email-masked]>; Tue, 18 Nov 1997 17:11:20 -0500 (EST)
>Received: from norman.cp10.es.xerox.com ([13.240.124.12]) by
alpha.xerox.com with SMTP id <56847(1)>; Tue, 18 Nov 1997 11:22:48 PST
>Received: from garfield.cp10.es.xerox.com by norman.cp10.es.xerox.com
(4.1/SMI-4.1)
>	id AA23312; Tue, 18 Nov 97 11:03:18 PST
>Received: from dellxpi-11 (dellxpi-11 [13.240.120.138])
>	by garfield.cp10.es.xerox.com (8.8.5/8.8.5) with SMTP id LAA03949;
>	Tue, 18 Nov 1997 11:00:05 -0800 (PST)
>Message-Id: <3.0.1.32.19971118110828.00f5d630@garfield>
>X-Sender: hastings@garfield
>X-Mailer: Windows Eudora Pro Version 3.0.1 (32)
>In-Reply-To: <[email-masked]>
>References: <5030300013707863000002L032*@MHS>
> <3.0.1.32.19971114121136.0107fb10@garfield>
>Mime-Version: 1.0
>Content-Type: text/plain; charset="us-ascii"
>

---

From: Jay Martin <[email-masked]>
Date: Fri, 21 Nov 1997 14:14:21 -0500

First, does anyone else on the Sense DL have any comments on this
topic?

I don't want this thread to degenerate down to just Tom Hastings
and myself.  Longstanding PWG participants know that Tom and I are
always at the opposite ends of the spectrum in defining things
(whatever the topic is), and I feel this discussion is going nowhere
fast without others' opinions.

> have sent jobs, the filter mechanism is just four bits.

If Sense is only for IPP--and it damn well better NOT be--then sure,
anyone can whip up some simple solution and get it out the door
tomorrow.

But Sense started off as a means to handle a *wide* variety of
events, with particular attention to the needs of network printers.
But not ONLY printers, since other system-related events would
also be desired (such as Job events, Queue events, etc.)

> separate events).

Ok, let's take a look at the set of "event classes" as currently

> 'processing-stopped' state.

Tom, here's where you and I *always* tend to disagree.  You stated
previously (in a prior posting) that a user demands *infinite*
flexibility in selecting which event classes to monitor.

I ask you: in REALITY, how many combinations are really viable here?

Here are some highly suspicious selections, ones that I don't
think are desired by anyone:

  a) Only "job-started-processing" events.  (Yeah, sure.)

  b) Only "printer-problems" events.  Why you would be using IPP
     to monitor printer problems is beyond me.  Sure, it can be done,
     but DON'T FORGET THAT THIS FACILITY IS ASSOCIATED WITH AN
     EXISTING JOB, and not some general-purpose printer-specific
     problem reporting mechanism.

I'll stop right there.  You see, when you talk about IPP events,
they are defined in terms of EXISTING jobs, and not some generic
"IPP printer" environment.  By definition (and I think you forget
this), IPP only gives you events after you've registered for them
as part of a job submission.  Am I wrong here?

I'm going to bet that there are only two combinations of event
classes that user's will consistently use:

  a) Give me all events, I'm a control freak

  b) Give me all events EXCEPT job-started-processing, since
     I'm not interested in intermediate events, only final
     events

Sure, some could argue for a couple more combinations.  But that's
just it--only a couple more.  Not 14 more to provide all possible
combinations.

Proposing that a N-way set of combinations is required is
ridiculous, IMHO, and makes me think you haven't really given
much thought about real world scenarios.

I'd really like to see some comments from others--either PRO or
CON--before continuing this discussion.

	...jay

---

From: Ron Bergman <[email-masked]>
Date: Mon, 24 Nov 1997 09:02:42 -0800 (Pacific Standard Time)

I certainly agree with Jay regarding the use of filters with SENSE.  It
should be possible to define a limited set of "Editions" that contain a
minimal amount of information that is not normally required.  For example,
when a user submits a print job he would like...

1. to be informed of any problems within the printer from the time the job
   is submitted, until the job has completed.  If the problem occurs
   before the job starts, the user most likely would like to be informed
   so he could take action to correct the problem.  If he doesn't care, he
   most likely does not want any job monitoring information or can simply
   filter this information locally.

2. when his job has started.

3. progress of his job while printing.

4. when his job has completed.

By filtering locally, the user application does not have to worry about
finding the correct "edition" and the SENSE "publisher" only has to create
one "edition" for the job.

I cannot see where this configuration is more complex or less ideal than
performing the filtering on the SENSE server.  Other than some additional
network traffic, which should be minor, I cannot think of any other
disadvantages.

Adding filters to the SENSE server for the few cases where the user does
not want the "complete package" does not appear to justify the added
complexity for the "publisher".  My guess is that in most cases where a
user does not want to know everything, he will ask for everthing and just
ignore what he doesn't care about.

	Ron Bergman
	Dataproducts Corp.