Re: Xap and serial

[Kevin Hawkins <lists@xxxxxxxxxxxxxxxxx>]

Hi Ian,

Bear in mind that the application you are using bridges all xAP traffic
to serial. In any fully bridged network transport you are restricted by
the maximum speed of the slowest network (quart into a pint pot
scenario). There are some things you could employ to improve this eg
compression etc and obviously any 'bridging' application should be
implemented with adequate buffers that can buffer transfers between high
and low speed networks. The same would be true of a simple115200 to 2400
baud inline buffer. Thus any burst would be paced appropriately onto the
lower speed network. A Tibbo therefore might not be the most suitable
device for this application. When I transfer Etherene originated
commands to serial C-Bus at 4800 baud the C-Bus serial interface buffer
space is only some 20+ characters so lots of buffering here. HomeVision
has only single command buffers - sometimes 2 bytes.

That aside I think it's important to first decide... is all that traffic
necessary in the first place ? Why have you got so many devices xAP
enabled and reporting status ( I assume) in HomeSeer - are they being
monitored elsewhere on xAP ? Then secondly why are you onwardly
transferring all this info to the serial port - are there devices on the
serial port that also always need this information.and if not why do you
send it ? There are various filters that can be employed to reduce the
traffic that is passed and it may be that you need to configure a
selective bridge (router) to filter unwanted information. You will know
which devices on the serial side are listening to which devices so you
only pass only the information that those devices need (filter on
source) - this can be further filtered by the class as you probably are
only interested in a particular schema. Additionally though you should
always pass any packets specifically targeted at devices sitting on the
serial link. This will reduce a whole lot of traffic - maybe 90% or
more. I tried this in an application with Crestron processors where the
Crestron processor registers with a 'router' the devices it is
interested in and the router then only forwards info from these devices.
Given your stats and assuming my 90% is a reasonable figure then you
have achieved a serial baud saturation of 15138 baud - very comfortable.
In fact given your stats you only need to see a 25% reduction but more
is obviously better.

If you do need really verbose realtime interaction between many devices
then xAP is probably not the most efficient protocol and there are other
bit stuffing protocols like 'SNAP' etc that truly compact this data. In
which case why not utilise SNAP or similar on the serial leg and bridge
this protocol to xAP with ample buffering. Implement a xAP to SNAP
conduit . What you gain in message size you will generally sacrifice in
message scope/expandability and certainly readability though.

You could (should you wish) also build on top of xAP a pacing layer
between devices - something that exchanges xAP packets between two
devices with many extra checks in place including extra error checking
,acknowledges, retries continuation packets etc - I guess a sort of
halfway house between XONXOFF and TCP. Indeed for some devices you
could even implement TCP tunnels but you have to be careful not to
compromise the methodologies of xAP. If you want 'guaranteed' delivery
use a custom schema that acknowledges receipt. These sort of features
may be formalised as part of a later xAP protocol update , maybe using
additional parameters within the header. We will not change or break the
existing protocol (it has remained solid for years now) but we can add
extra layers on top of it for those that want these type of features.
Being able to do such things is one of the great strengths of xAP and
you can do this now yourself anyway.

Re the Netiom - this is actually one of the fastest xAP devices around
and in an experiment I set up an output on a Netiom was monitored and if
it changed it was reversed by a PC application (xAP Mapper) and I was
achieving a throughput of over 6000 messages per minute - this was a
self pacing test - that is some going - and the limitation might still
have been the PC application rather than the Netiom . I have never known
the Netiom miss messages but I suppose if it was receiving very fast
Ethernet speed requests destined to output lots of serial data at a low
speed then if there is insufficient buffering something has to give.
Either the receiver (Netiom) has to throttle back the sender (XONXOFF
style) or data will inevitably be lost . Note it is still not missing
incoming data but it is unable to action its content fast enough. On the
receive side as Netiom is always transferring from low speed to higher
speed layers nothing can ever be lost. The other key aspect of the
Netiom is that it parses incoming xAP data inline / on the fly (unlike
other xAP libraries that reform xAP messages into structures for later
analysis), this is a different approach that embedded designers need to
implement - and indeed is something I am still learning with the
embedded xAP gateways I have coded. It's all to do with how long it
takes you to finish processing an incoming message and this can be quite
fast (unless a lower speed transport is involved). On a Netiom (barring
the serilal output port) it's pretty instant.

It is true that it certainly helps if xAP applications that are sending
updates on a lot of endpoints stagger these over a period, particularly
in BSC. Indeed on the Netiom you can configure it to step through each
endpoint one at a time over several minutes rather than burst them all
at once. When responding to a wildcarded xAPBSC.query command addressed
to every endpoint then it is important that all of these respond fairly
instantly as this provides a limited form of discovery. Off the top of
my head I suggestion in another thread at least 2ms per response which
gives a flood rate of 500 messages per second - but it may be that a
recommendation of longer than this is more desireable . Maybe 20ms
giving 50 messages per second ?

Regardless of any such a recommendation I think the key to your solution
is in choosing appropriate hardware to implement bridging. You need to
consider your peak xAP data arrival burst rates and the level of
filtering you can apply to reduce (filter) the number of messages to
transfer to the lower speed transport - and then ensure adequate buffers
are available or data will be (has to be ) lost. As such PC based
applications make excellent platforms for such applications. Otherwise
you need to code a 'bit mapped/low level protocol' to xAP conduit.

Kevin

Ian Davidson wrote:
>
> Hi
>
> It?s been quite a while since I did any development work with Xap and
> when I did it was only minimal. However I decide I would try to ease
> myself back into it as I get a few minutes.
>
> I mainly work with PIC?s but did want an Ethernet device. Rather than
> trying to put Ethernet directly into the Pic (I know it?s possibly but
> not by me!) I decide to use a Tibbo device to act as the Ethernet to
> serial gateway and just let the Pic talk serial.
>
> The Tibbo (www.tibbo.com) is quite powerful, but for the first tests I
> simply programmed it for UDP and to forward all traffic on port 3639
> to serial with a serial rate of 115200 Baud.
>
> In a terminal program I monitored the output and saw my Xap traffic
> happily flowing away. After a while and pausing the display it became
> apparent that some messages were broken. I then used the Xap viewer
> side by side with the terminal program and could see some missed
> messages. On closer inspection everything was fine until the Xap
> homeseer plug-in sent a large amount of Xap messages. Looking more
> closely at this homeseer actually sent out 69 Xap messages in 0.84
> seconds. The average message length was 222 bytes which equates to
> 15318 bytes. To send this serially would require a start and stop bit
> as well as the eight data bits which equates to 153180 bits in 0.84
> seconds or 182357 Baud.
>
> This is why I was missing and getting some corrupt messages. I know
> originally Xap was aimed at a multitude of devices but this rate of
> output clearly rules out almost all serial devices. Even if the Pic
> did have a direct Ethernet interface it would be pushed to cope with
> this rate of data. I have the Xap Netiom which is great and I know
> uses a Pic with direct Ethernet but I?m sure it would not be able to
> cope with the high data rate at times. Although I have not had an
> issue with my netiom this is probably due to the commands to it not
> being sent during one of these data storms.
>
> I?m really just looking for comments on this I would like to use Xap
> on Pic based devices but I would also like to guarantee it didn?t miss
> any messages and as it stands I can not. I?m looking to control some
> devices and having used Xap, which is great and still growing would
> have seemed ideal but at the moment it seems as though a lighter
> protocol would probably be more beneficial. Can anyone think of a
> solution, is it possible to limit the speed of Xap packets onto the
> network or would this start to limit and slow other Xap PC based apps.
>
> Thanks
>
> Ian
>
>