Supported Sensors

FLIR / Point Grey GigE

pointgrey-gige
V2.0.0-b.7
Ethernet
230

Supported hardware versions

The driver interface communicates with the camera sensor using the FlyCapture SDK for Blackfly GigE Vision cameras.

Manufacturer PointGrey
Hardware ID Driver supports all GigE Cameras, tested with the Blackly-BFLY-PGE-23S6C
Firmware version(s) supported Firmware: 1.40.3.1
SDK version supported FlyCapture SDK: 2.8

Sensor background and requirements

FLIR recommends the Blackfly GigE cameras for automotive purposes due to the high frame rate and resolution that is provided.

Flexible mechanical design, compatibility with third party peripherals, and support for industry standard protocols allow FLIR cameras to excel in industrial camera applications.

PolySync likes to use the 2.3 MP global shutter, and 2.3 MP rolling shutter cameras on the sensor mule vehicles.

Sensor notes

Hardware requirements

  • Compatible lens
    • Visit FLIR, or their recommended lens outfitter Edmund Optics to find a compatible lens for the Blackfly camera
  • 12V power supply
    • AC power supplies that produce 12V can be used for bench testing, and can be purchased from FLIR
  • Gigabit Ethernet connection to an ECU, or network switch
    • It’s strongly recommended to provide a direct connection from the camera to the ECU
  • ECU with PolySync Core installed

Configuring the sensor

Using the Windows version of the Flycapture2 application you can find the IP address of new Blackfly sensors. Once the sensor is detectable with FlyCapture2, you can use the GigEConfigurator application (Windows only) to set a static IP address.

Next the sensor can be connected to the Linux ECU with PolySync Core. The ECUs Network Interface Card (NIC) needs to be configured to have an IP address on the same subnet as the Blackfly camera.

Setting the cameras static IP address

To set the IP address continue reading, or follow the FLIR guide and skip to the next section.

On a machine running Windows 7 or newer:

  1. Visit FLIR download center to download the required software (login required)
    • Flycapture2: recommended to download Windows and Linux versions
    • GigEConfigurator: Windows support only
  2. Install both software applications for Windows
  3. Connect an Ethernet cable between the camera and the Windows machine
  4. Power the camera
  5. On the ECU, set a static IP address within the 169.254.X.X subnet
  6. Start the GigEConfigurator software
  7. Select the desired camera from the tree on the left side, you should see the camera model and serial number if the camera is detected
  8. Near the bottom of the screen, set a static IP address and subet
  9. Click the Set IP Address button
  10. At the bottom of the screen, click the Copy from current button
  11. Ensure the DHCP checkbox is unselected
    • Only the LLA checkbox should be selected
  12. Click the Set Persistent IP Address button

Power cycle the sensor and ensure you can ping the sensor at the new static IP address, after updating the ECUs IP address to match the new subnet.

Configuring the ECU

Linux network set up

The Ethernet network must be properly configured before PolySync or FLIR FlyCapture software can communicate with the sensor on the target ECU.

It’s highly recommended to use point-to-point Ethernet communication between the sensor and the ECU to minimize dropped frames and lower the latency. You may connect the sensor to a switch and be prone to high latency and dropper frames.

Setting the ECUs static IP address

Once the cameras IP address is known, set a static IP address for the ECUs NIC in the same IP subnet range as the camera.

For example if the cameras IP address is 192.168.25.2, it’s said to be on the 192.168.25 subnet, and the ECU should be configured with IP address 192.168.25.X.

Increase Linux internal network buffer sizes

This is an optional step, however if you’re experiencing dropped frames with your current configuration this modification of the default configuration is highly recommended.

PolySync provides a configuration file in the /etc/sysctl.d/10-polysync.conf file. Modify this file to match the values below:

$ sudo vim /etc/sysctl.d/10-polysync.conf
net.core.rmem_default = 1048576
net.core.rmem_max = 1048576
net.core.wmem_default = 1048576
net.core.wmem_max = 1048576
net.core.netdev_max_backlog = 30000

Restart your system, or issue the following command to ensure the recent changes are loaded into the system.

$ sudo sysctl -p /etc/profile.d/10-polysync.conf

Configuring the PolySync driver

Performance is heavily affected by networking and system-level settings. The default PolySync Configurator configuration has shown to work out of the box on most systems.

Adding the sensor to the SDF

Using the Configurator tool, add a sensor node to the SDF.

The Blackfly ‘Node Interface’ name is pointgrey-gige.

Parse the sensor identifier

Once the node is defined in the SDF, and the Ethernet connection has been properly configured and established, the dynamic driver is able to query the connected cameras for their unique serial number identifier.

PolySync users this value to locate and connect to the camera. The sensors identifier is stored in the SDF as parameter Sensor 0 Identifier.

To parse the sensor identifier of all connect Blackfly cameras, find the node ID of the dynamic driver and pass the -l flag. This is the expected output when one camera is connected.

$ polysync-core-dynamic-driver -n 3 -l 
2017-04-20 15:48:16.04s DEBUG  [pointgrey-gige-] - build version 2.0.13-1489441994
2017-04-20 15:48:17.04s DEBUG  [pointgrey-gige-1688863901464412] - created participant - GUID: 0x0006000344EBB75C d(1688863901464412)
2017-04-20 15:48:17.04s DEBUG  [pointgrey-gige-1688863901464412] - message types visible to this node: 47
2017-04-20 15:48:17.04s DEBUG  [pointgrey-gige-1688863901464412] - transition to state: INIT - timestamp: 1492728497677217
2017-04-20 15:48:17.04s DEBUG  [pointgrey-gige-1688863901464412] - frame PLATFORM (4) - origin (0.000,0.000,0.000) - orientation (0.0000,0.0000,0.0000)
2017-04-20 15:48:17.04s DEBUG  [pointgrey-gige-1688863901464412] - using publish encoder -- src_fmt: 10 - w/h: 640/480 - publish_fmt: 5 - w/h: 320/240

default H264 encoder configuration
  plugin name: 'x264enc'
  profile: 9
  speed-preset: 7
  tune flags: 0x0
  bitrate: 2048

listing all available PointGrey GigE devices
found device SN: 15395510 - IP 192.168.10.2 - MAC: 00:B0:9D:EA:EA:B6

2017-04-20 15:48:19.04s DEBUG  [pointgrey-gige-1688863901464412] - transition to state: OK - timestamp: 1492728499460537
2017-04-20 15:48:19.04s DEBUG  [pointgrey-gige-1688863901464412] - exit signaled - terminating

Notice the fonud device SN: 15395510 - ... line. This SN must be entered as the Sensor 0 Identifer in the SDF using the Configurator.

Configuration parameters and calibrating the sensor

The image data can be configured at three locations throughout the image processing pipeline.

  • The source image is requested from the image device
    • When it’s configured to use shared memory, each source image buffer is copied to the shared memory segment
  • The logged image is written to a binary plog file that can be decoded and reprocessed by the dynamic driver
  • The published image is sent to the global PolySync Ethernet bus, and is usually a highly compressed diagnostic image stream used to validate the sensor is working properly, but not used for developing algorithms
Default configuration
Image width x height Pixel format Frame rate Notes
Source image 640×480 RGB24 30 FPS ROI X/Y offsets at 0,0
Logged image 640×480 RGB24 N/A
Published image 320×240 H264 30 FPS

Modify each of these parameters in the Configurator table.

The maximum resolution and frame rate for each camera can be found in the respective user manuals, or using the FlyCapture software.

For a list of available pixel formats, check the C data model API documentation.

Region of Interest

You can set the ROI to center the image, or view/access the desired pixels for all frames.

The Region of Interest (ROI) in the FlyCapture SDK is equivalent to the PolySync X and Y pixel offsets, parameters 1086 and 1087 respectfully.

To change the camera’s Region of Interest (ROI), update the Video Device 0 Source X pixel offset and Video Device 0 Source Y pixel offset parameters in the nodes Configurator entry.

You can calculate the X and Y pixel offset using the FlyCapture tool provided by FLIR.

  1. Start the FlyCapture software on the Linux machine
  2. Once FlyCapture software is running select the camera you want to configure and click the “Configure Selected” button
  3. Go to the thrid tab, “Custom Video Modes”
  4. Click the “Center ROI button”
  5. Record the Left (X) and Top (Y) values which correlate to the X pixel offset, and Y pixel offset
  6. Open the Configurator and update the two parameters for the Blackfly GigE node
    • X pixel offset: parameter ID 1086
    • Y pixel offset: parameter ID 1087

FlyCapture2

Configure inter-packet delay

If you are experiencing continuous dropped frames on the subscriber-side, like the image-data-viewer or Studio video plugin, it’s likely you need to increase the inter-packet delay.

This parameter can be set using the Configurator. It’s parameter ID is X and it’s located in the ‘FlyCapture GigE Configuration’ section, near the bottom of the table.

Inter-packet delay in the Configurator

If the inter-packet delay is maxed out and you’re still seeing dropped frames you’ll need to reduce the frame rate, or reduce/optimize your network configuration.

Video Mode Descriptions

This table is taken directly from the BlackflyTechnicalRefernce.pdf document provided by FLIR. If you’re unsure what different video modes do, it is best to keep the default configuration.

Blackfly video mode descriptions

Shared memory configuration

This is an optional step. Shared memory queues provide drastically faster read/write speeds than the global shared PolySync bus that exists on the Ethernet wire.

Shared memory segments exist to allow two or more host-local nodes to access the data. Commonly the data is very large, such as uncompressed image data that would normally saturate a Gigabit Ethernet bus.

  1. Configure the PolySync driver to use the shared memory queue by providing a unique shared memory queue key
    • The driver manages the creation and cleanup of the memory segment
  2. Configure your custom application to read data by referencing the unique shared memory queue key
  3. Create the shared memory segment
  4. Start the PolySync driver and custom application

If the shared memory buffer is not set up when the dynamic driver starts, it will detect this and populate an error in the polysync.log indicating the appropriate buffer size that needs to be configured.

Validating the sensor is properly configured

If you’re approaching a new PolySync system or need to validate an existing configuration you can use the following checklist to ensure the sensor is properly configured.

Set up checklist

If the sensor passes these checks then the PolySync dynamic driver will likely be able to communicate with the camera sensor.

  1. Ensure you can ping the sensor at the static IP address
    • Get the IP address from the Configurator node entry, static IP address set in the preparing a sensor section
      • $ ping xxx.xxx.xxx.xxx
      • 64 bytes from xxx.xxx.xxx.xxx: icmp_seq=1 ttl=64 time=0.030 ms
      • ...
  2. Ensure the camera is listed when you use the dynamic driver Blackly interfaces -l flag
    • Get the Blackfly node ID from the Configurator, and temporarily start the dynamic driver on the command line
  3. Ensure the serial number printed from the previous step match the Sensor 0 Identifier parameter in the Configurator
  4. If the camera is configured to use shared memory, ensure the shared memory segment has been allocated from Linux’s perspective
    • $ ipcs -l

Starting the PolySync driver

The configuration set in the Configurator is loaded from the SDF when the dynamic driver starts. It connects to the sensor over the Ethernet interface, requests the data, and waits for confirmation that the sensor configuration is valid.

When the dynamic driver receives the first full frame of data it begins processing the data, abstracting the data from the OEM data structure in a high-level hardware agnostic message type. In this case the cameras data is placed in a ps_image_data_msg.

  1. Power the sensor and ECU on
  2. Optionally follow the set up checklist
  3. Start the PolySync Core manager
    • $ sudo service polysync-core-manager start
  4. Start the dynamic driver process

Starting the node manually on the command line

To start a dynamic driver node on the command line, the node must first be defined in the SDF using the Configurator application.

Each node defined in the Configurator has a unique node ID which points to the nodes configuration. This article explains how to find the node ID.

Command line flags and usage

Once the node ID is known (substitute for X), the dynamic driver node for the supported sensor can be started with the base command:

$ polysync-core-dynamic-driver -n X

Each sensor supports an array of command line arguments. To see a full list of command line arguments, pass the -h help flag:

$ polysync-core-dynamic-driver -n X -h  |  less

There’s a lot of output so we recommend you pipe the output to less, but it’s not required.

Flag Required Description Arguments
-e No Export a JSON support string describing the interface, used by the SDF configuration tool N/A
-h No Show the help message N/A
-i <pal.so> No Use provided PAL interface file instead of what is stored in the SDF Path to the dyanmic driver interface PAL shared object library
-l No List all available PointGrey GigE devices N/A
-n <N> Yes SDF node configuration identifier for the node SDF node ID from the Configurator, [0-65536]
-o No Reserved N/A
-O No Check the node SDF configuration for required updates and exit option, returns exit status zero if no changes are required
-p <file.plog> No Use provided logfile in Record and Replay operations instead of the default File path to a PolySync plog logfile
-r <N> No SDF runtime configuration key that specifies the domain to operated under, the default domain is used otherwise Runtime configuration key, [0-65536]
-s <psync.sdf> No Use provided SDF instead of the default File path to an SDF file
-u No Allow updates to the SDF configuration during the normal runtime if needed (does not exit) N/A
-U No Update the node SDF configuration and exit N/A
-w No Disable the hardware interface(s), allowing the node to run without hardware connected - also known as replay mode N/A
DTC codes and common fixes
DTC value DTC name Fault description Notes
304 DTC_NOINTERFACE Interface not available Activated when the sensor is not reachable at the IP address set in the Configurator; activated when the sensor becomes unreachable during runtime

Accessing sensor data

When the dynamic driver node is operating in an OK state then data is being published to the global PolySync bus, and any node can subscribe to the high-level message type(s) output by the dynamic driver node.

There are several tools that PolySync provides to quickly validate that data exists on the bus.

Access sensor data with PolySync nodes that subscribe to the sensor’s output message types.

Input / output message types

The FLIR Blackfly dynamic driver node outputs the following message types to the bus.

Message API Docs Notes
Publishes pre-defined ps_image_data_msg Sensor Data Model Publishing is disabled when the published pixel format parameter is set to PIXEL_FORMAT_INVALID in the SDF Configurator
Publishes shared memory message type PSYNC_SHDMEM_MSG_TYPE_IMAGE_DATA Shared memory Publishing is disabled when output shared memory queue key is set to value 0

By default the ps_image_data_msg is published to the bus when you create a new Blackfly node. You can enable and disable the publishing of specific message types in the Configurator.

Image data message fields

Data type Name Description Message field populated by this sensor
ps_msg_header header PolySync message header. Yes
ps_sensor_descriptor sensor_descriptor Sensor descriptor. Yes
ps_timestamp timestamp Image timestamp. Yes
ps_native_timestamp native_timestamp Native timestamp for the image. Provided by some devices. Check ps_native_timestamp.format for meaning. Format value PSYNC_NATIVE_TIMESTAMP_FORMAT_INVALID means not available. Yes
ps_pixel_format_kind pixel_format Image data format. Yes
ps_identifier frame_id Image counter. Value PSYNC_IDENTIFIER_INVALID means not available. Yes
DDS_unsigned_short width Image width. [pixels] Yes
DDS_unsigned_short height Image height. [pixels] Yes
DDS_sequence_char data_buffer Image data buffer. Yes

Filtering incoming data for this sensor

An application that subscribes to a given message type is able to see data from more than one sensor or source.

Applications can filter for specific sensors and data sources in the message callback in C applications, or the messageEvent in C++ applications.

Filter incoming messages for this sensor with ps_sensor_kind value 230.

You can find all sensor descriptor values in this article.

Resources and configuration tools

This section has supporting resources and tools that are referenced throughout the article.

Connecting to a Blackfly camera with FlyCapture

On a machine running Windows 7 or newer:

  1. Visit FLIR download center to download the required software (login required)
    • Flycapture: recommended to download Windows and Linux versions
  2. Install the FlyCapture software application for Linux Ubuntu machines
  3. Connect an Ethernet cable between the camera and the Ubuntu machine
  4. Power the camera
  5. Set a static IP address in the same subnet as the sensor
    • A static IP address is set in the [Preparing the sensor]{#sensor-preparation} section
  6. Start the FlyCapture2 software
    • $ flycap

FlyCapture

FlyCapture shows the list of detected GigE sensors that are connected to the ECUs network. If no devices are listed, double check the network configuration.

Select the desired image device and click the “Configure Selected” button.

You can configure each camera individually. Some values, like the ones below, can be calculated and captured by FlyCapture software then entered in the PolySync Configurator tool for persistent runtime configuration.

  • ‘Custom Video Modes’ tab
    • Region of interest / X/Y pixel offset
    • Maximum GigE inter-packet delay value
    • Determine valid operation modes

Hardware synchronization trigger

PolySync offers support for an external triggering device using the Teensy++ 2.0 USB board that enables you to synchronize two or more camera streams.

Submit a request to the support desk at help@polysync.io and PolySync can provide the firmware. The firmware is not proprietary, in fact any 3-5 volt signal can be used.

Interface Specification Docs

The Flycapture2 SDK must be requested and downloaded directly from FLIR. The SDK contains detailed documentation and defines the communication protocol used by the PolySync Core dynamic driver to communicate with the sensor.


Useful FLIR Resources