Packetlogger update with fixed field lengths and lookup values

The NMEA has in the past decided to keep the NMEA 2000 PGN database private. To get access to the database you need to become a member of the NMEA or buy the standard outright. Appendix B is $900. And, as of recent, this is not enough as the NMEA is now desupporting PGNs in favour of SAE standards that require you to buy SAE standards as well.

The money for the standards in itself is not so much a problem as is the fact that if you buy it you are not allowed to publish the data contained in the standard.

The best unencumbered source for NMEA 2000 data that I have found so far is the NMEA 2000 database in Vector's CANalyzer / CANoe demo product that you can download here. It does seem to contain a slightly older version of the NMEA database as not all fields described by the NMEA are present.

It is a pity that the various industry bodies keep insisting on non-free standards. It is my belief that open standards such as the Internet standards which are published by the IETF using the RFC process form a better way to promote standard adoption.

Anyway, using the data mentioned above I have been able to confirm a lot of guesses that I had already made earlier, and fix some other fields. In particular the lookup values of a lot of fields have seen significant improvements. I have also been able to determine field lengths for PGNs and fields that I do not have logs for.

The analyzer now does correct decoding of everything that I have seen in the field through my own logfiles and those that have been sent to me, as far as I can see.

As usual, you can download the NMEA analysis + reader programs here.

The complete list of improvements:

• All field lengths of NMEA PGNs should be correct.
• All field descriptions of NMEA PGNs with lookup values should be correct.
• The XML file now also mentions whether a field is signed or unsigned.
• The XML file now describes the real NMEA 2000 field type, not the converted type. For instance, the analyzer converts radians to degrees as these are more 'natural' for humans to understand. Previously those fields were described as being in 'degrees', now the correct unit is given as 'rad', and the resolution is also given with the correct magnitude.
• The analyzer now correctly decodes signed integer fields such as the time offset in minutes in PGN 129033.
• The database now knows about IEEE float data fields. The analyzer cannot print these out yet as I have no samples of this data yet.
• Initial support for Darwin (Mac OS X). The analyzer works very well. I've ported the serial port reader programs as well, but the actisense-serial program does not seem to want to read from the serial port as presented by Actisense's OS X drivers yet.

As of now I am no longer in need of unqualified log files. You still can and should send me a log file if you see a PGN that the analyzer decodes incorrectly, of course.

Code for converting a CAN message ID to ISO 11783 / NMEA 2000 fields

I've received a number of requests from people who are trying to get microcontrollers up & running that they want to attach to the N2K bus.

Apart from the logical content of the datafields for each PGN, one of the other things that you will end up doing is filtering out the priority, PGN, source address and destination address from the underlying CAN message.

NMEA 2000 messages are always CAN 2.0B messages with the "Extended Frame" format. It uses the ISO-11783 standard as the physical layer. That standard is where we get the "PGN" term from.

Once you get a ISO-11783 CAN message ID from your device, it will contain an encoding for the PGN number, the priority, the source ID and possibly a destination address. To save bits the PGNs that are not addressable imply a "broadcast" destination of 255. That format is called PDU2. Other PGNs are addressable, and contain an explicit 8 bit destination in the PS field. That format is called PDU1. If you want to know more, try googling for ISO11783 PDU1 for more information.

Once you've got the 32 bit CAN message ID (which you should be able to get from your hardware device pretty easily) you need to convert the message ID to priority, PGN, source and destination. Here is the source code that I came up with to do this:

static void getISO11783BitsFromCanId(unsigned int id, unsigned int * prio, unsigned int * pgn, unsigned int * src, unsigned int * dst)
{
  unsigned char PF = (unsigned char) (id >> 16);
  unsigned char PS = (unsigned char) (id >> 8);
  unsigned char DP = (unsigned char) (id >> 24) & 1;

  if (src)
  {
    *src = (unsigned char) id >> 0;
  }
  if (prio)
  {
    *prio = (unsigned char) ((id >> 26) & 0x7);
  }

  if (PF < 240)
  {
    /* PDU1 format, the PS contains the destination address */
    if (dst)
    {
      *dst = PS;
    }
    if (pgn)
    {
      *pgn = (DP << 16) + (PF << 8);
    }
  }
  else
  {
    /* PDU2 format, the destination is implied global and the PGN is extended */
    if (dst)
    {
      *dst = 0xff;
    }
    if (pgn)
    {
      *pgn = (DP << 16) + (PF << 8) + PS;
    }
  }

}

Packetlogger update: PGN list complete, Linux version available

I'm back from finishing a boat, testing it by sailing to Denmark and then showing it at the HISWA boat show.

Now is the time to catch up with my backlog here. Let's start off with an update to the packetlogger utility. The new 'Summer 2010' release out now has a ton of improvements.

You can download the packetlogger here.

The improvements made are:

• The code now knows about all official PGNs as the NMEA has released the complete list of PGNs and field descrioptions. Unfortunately, that list does not contain field sizes, lookup values or other information but it still helped a lot with my understanding some of the PGNs.
  
• Linux binaries are included for x86. It was compiled on a Debian 5 release, so it should run on all recent Linux distributions. The Linux port also includes an Actisense NGT-1 reader, which was developed with the kind support of Actisense.
• The code now understands PGNs with repeating fields a lot better. Some PGNs have a variable list of fields, with the last few fields repeating a number of times.
• Add a json option to the analyzer so the output can be fed to web-oriented languages such as Perl, PHP and Javascript with ease.
• Log the manufacturer for all manufacturer specific PGNs.
• Remove explicit empty PGNs that we only knew the manufacturer for.
• Verified that the Mastervolt NMEA 2000 interface is compatible.

This should keep many folks happy for a while. Keep the feedback coming, by private email is fine if you don't want to show up here.

Packetlogger for Linux

I've ported the packetlogger program to Linux. At the moment the only interface that I have hooked up is the Lawicel CAN-USB, as I plan to use the Actisense interface on Microsoft Windows and thus have no need (yet) for it to run on Linux.

It turns out that the Lawicel (like the Actisense) uses an FTDI chip to interface to USB. I could not get the FTDI D2xx libraries to work properly on Debian (Lenny). The workaround was simple: interface to the pseudo serial port directly, that works fine and is just about the same amount of work. It also means there is less interference of additional layers of software. My code uses a separate process to do the reading anyway, and uses the UNIX pipe mechanism to transport the data to further processing layers.

Now I shall have to start implementing the real application, which is storing the data permanently in a database on a USB stick and upload the same data to a server on the Internet. It will have to use some sort of configuration file that tells the code what data needs to be stored, and how often.

Something like:

local:
heading: every 300 seconds or 10 degree course change
location: every 300 seconds or 100 m change


internet:
location: every 300 seconds or 100m change