The DAVIC Section Filter API

MPEG-2 private sections are not just limited to containing service information – they can contain just about any type of data that the broadcaster wants to put there. DSM-CC object carousels are broadcast in private sections, and a broadcaster can also use private sections to transmit their own proprietary filesystem format or any other private data. Of course, if broadcasters do this, then applications need a mechanism for actually getting at the data that’s contained in MPEG-2 sections.

The section filtering API is contained in the org.davic.mpeg.sections package, and provides several different ways for applications to filter the private sections that interest them from a transport stream. Each of these ways is designed to cope with a specific type of situation where an application may want to access private sections.

First, let’s review the format of an MPEG-2 private section:

The section_syntax_indicator field shows whether the section is a private section or not. If this field has the value ‘0’ then the data within the section is in a completely proprietary format. A value of ‘1’ indicates that it follows the MPEG-2 private section format, and it is these sections that are of most interest to the section filtering API.

Given that filtering sections may be a time-consuming operation (because we don’t know when the sections that match our criteria will arrive), it’s no great surprise that section filtering is an asynchronous operation. The application simply sets up some filters, sets a listener for the events that get generated when a matching section is found, and then waits for events to start arriving. These filters can be set on a number of criteria, from simply filtering all sections on a specific PID, to filtering all sections with a specific table ID ona given PID, to filtering based on other values in the section header or the section data. The section filtering API allows applications to use both positive and negative filters, so an application can either filter for sections that match a specific patter, or for those that do not meet a given pattern. Or course, the API can also do basic things such as simple filtering based on the table ID, for instance.

Life isn’t really that simple, of course, but you’ll be pleased to hear that it isn’t actually much more complicated. The main complication is the fact that section filters are scarce resources (usually very scarce resources), and so applications have to take great care to manage section filters correctly and not use scarce resources when they don’t need to.

Most receivers will support a number of hardware section filters (up to 32 in most modern DTV chipsets). Many of these will be used by the middleware for parsing service information or DSM-CC object carousels, but a number of section filters are guaranteed to be available to applications. While it is possible to carry out section filtering in software, the performance impact of doing this is usually too high for it to be a real option. Software solutions risk of slowing down the receiver unacceptably, and potentially missing sections that you are supposed to be filtering, making hardware solutions much more attractive. The section filtering API can be used with either hardware or software section filters, and in practise it adds a number of features that are built on top of what is offered by hardware section filters.

API description

The API features four main classes. The first of these is the Section class that represents an MPEG-2 section and which allows access to the section data. This class provides method for accessing all the various section headers, as well as the section data. There is nothing really complicated about this class, and so we will not describe it here in any detail.

The section filtering API is another API that uses the resource notification API to help control resource use amongst the various applications that use it. In this case, the API uses a modified form of the resource notification API design pattern which we shall see later.

Section filters

The SectionFilter class represents a real section filter, which may be implemented by the receiver in hardware or software.

public abstract class SectionFilter {
  public void startFiltering(
    java.lang.Object appData,
    int pid,
    int table_id,
    int offset,
    byte[] posFilterDef,
    byte[] posFilterMask,
    byte[] negFilterDef,
    byte[] negFilterMask);

  public void setTimeOut(long milliseconds);
  public void stopFiltering();

  public void addSectionFilterListener
    (SectionFilterListener listener);
  public void removeSectionFilterListener
    (SectionFilterListener listener);
}

We only show the most complex version of the startFiltering() method here – there are six versions in total, each one having some of the arguments listed in the most complex version above.

The startFiltering() and stopFiltering() methods start and stop the processing of a transport stream, as you’d expect. However, there are some restrictions on when a section filter can be started. If the section filter is attached to a transport stream (see the section below on section filter groups), then the filter will begin filtering immediately. Otherwise, it will simply store these settings for when the section filter is attached to a transport stream.

The parameters to the startFiltering() method are as follows:

appData

application-specific data that uniquely identifies this filter to the application. This is used to identify section filtering events in a way similar to that used in the service information API, so that a single listener can accept events from more than one section filter and still be able to distinguish between them

pid

the MPEG PID of the stream that filtering should be carried out on

table_id

the value of the table ID for sections that should be filtered

offset

an offset into the section that indicates the part of the section upon which the filter should operate. This is a value between 3 and 31 that is designed to skip the section length and the table ID values, but which allows the first part of the section data to be filtered.

posFilterDef

the values which the filter should match. The exact parts of the value that are considered important are determined by the filter mask. A positive filter is triggered when the parts of the filter definition that are enabled by the filter mask match the contents of the section, starting at the appropriate offset into the section.

posFilterMask

a bit mask indicating which bits in the section should be matched to the filter definition

negFilterDef

a filter definition for negative filtering. Negative filtering causes the filter to be triggered when the parts of the filter definition that are enabled by the filter mask do not match the contents of the section, starting at the appropriate offset into the section.

negFilterMask

a bit mask for negative section filtering. This serves the same purpose as the positive filter mask, but is applied to the negative filter definition instead.

As we can see, the section filtering API supports both positive and negative filtering, and actually allows both types of filtering to be carried out at the same time. The setTimeOut() method allows the application to specify that filtering should end if no sections are matched within a specified period.

The section filtering API is an asynchronous API as we’ve already mentioned, and the final two methods allow an application to register and unregister listeners for events from this particular filter. These may indicate that a new section has arrived, or they may indicate a problem with the section filter or a change in its state. We will take a closer look at the events that may be generated later in this section.

Section filter types

There are three types of section filter available to an application, each designed for a different purpose and with different strengths and weaknesses.

The first of these is the simple section filter. This is implemented by the SimpleSectionFilter class, and is simply a one-shot filter that filters a single section before stopping. When a section is matched, filtering stops and a SectionAvailableEvent is generated. An application may then get a reference to the filtered section by calling the SimpleSectionFilter.getSection() method.

A simple section filter must be restarted every time a section is filtered if the application wishes to filter more than one section. This is a little inconvenient if you want to filter a stream of sections, and so the simple section filter is best used in those cases where only a few sections are likely to be filtered, and where the gap between sections in the stream is long enough to allow the filter to be reset.

For those cases where many sections have to be filtered from the stream, the ring section filter is a better choice. This is implemented by the RingSectionFilter class. It provides a circular buffer of sections (where the size of the buffer is specified when the filter is created) and where more than one section can be stored before the filter will stop.

Whenever a new section is matched, it is placed in the next free slot in the buffer and a SectionAvailableEvent is generated. An application may then read the buffer using the RingSectionFilter.getSections() method.

Before a slot in the buffer can be re-used, the Section.setEmpty() method must be called on the section that occupies that slot. When no more slots in the buffer are available, filtering will stop. For this reason, it’s important that the application calls Section.setEmpty() as soon as possible after the section is received.

This process is shown in the diagrams below:

Ring section filters can store multiple sections.

Applications should call setEmpty() on a section to free a slot for re-use by the section filter.

The section filter can only operate when there is an empty slot to put a newly-filtered section.

Newly-filtered sections are stored in the next available slot.

1. A ring section filter can contain more than one section at the same time.
2. Each slot in the buffer must be freed before if can be re-used by calling setEmpty().
3. Any new sections will be stored in the next free slot, until every slot is full.
4. When every slot in the buffer is full, the section filter will stop automatically.

The final type of section filter that is available to applications is the table section filter. This allows applications to filter out entire service information tables easily. It is implemented by the TableSectionFilter class, and automatically defines a buffer big enough to contain the entire table. As with the RingSectionFilter, a SectionAvailableEvent is generated after every new section is received. When the entire table has been received, filtering will stop. Unlike the RingSectionFilter, however, the application doesn’t need to call Section.setEmpty() as soon as possible after a section is received.

Since a new version of a table may be broadcast while it’s being filtered, only the first version of any table will be filtered. If the version number changes while filtering is in progress, a VersionChangeDetectedEvent (see below) will be generated. Sections with the new version number will not be matched.

Section filter groups

Section filters are a scarce resource, and so the SectionFilterGroup class provides a way for the receiver and the application to manage section filter resources. It also allows for several section filters to be grouped together (hence the name) for improving performance and ease of use. Section filter groups can be used to prevent deadlocks by reserving several section filters in an atomic operation, so an application that needs two section filters for a given task can be guaranteed to receive either both of them or none of them. This prevents the case where competing applications block each other from gaining access to the filters that they need.

Section filter groups are implemented by the SectionFilterGroup class:

public class SectionFilterGroup
  implements ResourceProxy, ResourceServer {

  public SectionFilterGroup(int numberOfFilters)

  public void attach(
    org.davic.mpeg.TransportStream stream,
    ResourceClient client,
    Object requestData)

  public void detach()

  public org.davic.mpeg.TransportStream getSource()

  public SimpleSectionFilter newSimpleSectionFilter()
  public SimpleSectionFilter newSimpleSectionFilter(
    int sectionSize)

  public RingSectionFilter newRingSectionFilter(
    int ringSize)
  public RingSectionFilter newRingSectionFilter(
    int ringSize,
    int sectionSize)

  public TableSectionFilter newTableSectionFilter(
    int sectionSize)

  public void removeResourceStatusEventListener
    (ResourceStatusListener listener);
  public void addResourceStatusEventListener
    (ResourceStatusListener listener);
}

As we can see if we look at the SectionFilterGroup class, it implements both the ResourceProxy and ResourceServer interfaces from the resource notification API. In this way, the section filter API is slightly unusual, although it’s no harder to use than any other API which implements the resource notification API.

Section filter groups are created via the public constructor, which takes an integer argument that specifies how many filters are in that group. By allowing applications to group filters like this, it makes it easy to reserve more than one filter at a time and so reduce the potential for deadlocks among applications that use section filters.

Once we have a SectionFilterGroup that contains the number of section filters we need, we can create individual section filters within that group. The SectionFilterGroup class allows us to create the type of section filter that is most suitable for the purpose we want by a call to the appropriate method.

Before a section filter can be used, we have to attach the group that contains it to the transport stream that we wish to filter in. The attach() method attaches a section filter group to a particular transport stream. At this point, the API reserves all the section filters needed (if they are available) and attaches them to the transport stream. When the section filter group is attached to a transport stream, all the section filters that it contains will automatically be started. An application can release the resources used by a section filter group (and stop any active section filters) by calling the detach() method.

Section filters are a scarce resource, and only two section filters may be available for all of the applications in a given service. For this reason, applications need to be careful how they use section filter groups and when they attach and detach filters. In general, section filter groups should contain the smallest number of section filters needed for the task at hand. If an application is carrying out two tasks using section filters, there should be two section filter groups in order to make sure that filters are only reserved when they are actually being used. The only exception to this is when you can design your filtering criteria to re-use a single filter for more than one task. Similarly, applications should attach a section filter group to a transport stream immediately before they want to start filtering, and detach it as soon as they are done. A filter group which is attached to a transport stream but not started may be preventing other applications from using those filters.

Section filter events

The final piece of the section filtering puzzle is the SectionFilterListener interface. This is implemented by the application in order to allow it to receive events from the section filters about sections that match the requirements.

The possible events that an application can receive are:

• IncompleteFilteringEvent – the section filter can’t perform any filtering because of an incorrectly defined set of filter parameters
• TimeOutEvent – no sections have been filtered within the specified timeout period, and so the section filter has stopped itself
• SectionAvailableEvent – a new section has been filtered and is available from the filter. The application should call the getSection() method (for a simple section filter) or getSections() method (for a ring or table section filter) to get the new section data.
• VersionChangeDetectedEvent – the version number of the sections has changed. The filter will ignore sections with a different version number to previous sections that have been filtered, in order to make sure that there are no inconsistencies in the filtered data.

An example

The following example shows how an application may use a section filter. The first piece of code shows how the application would set up the section filter and activate it:

// Create a SectionFilterGroup that reserves two hardware
// section filters
SectionFilterGroup filterGroup =
  new SectionFilterGroup(2);

// Create a couple of section filters within the filter
// group.  First, we'll create a simple section filter
SimpleSectionFilter simpleFilter;
simpleFilter = filterGroup.newSimpleSectionFilter();

// Now we create a ring section filter.  Our ring section
// filter will hold up to five sections, which gives us a
// little leeway in the time it takes to process a
// section
RingSectionFilter ringFilter;
ringFilter = filterGroup.newRingSectionFilter(5);

// Since the section filter isn't much use to us if we
// don't know when we've got a new section, we register
// an event listener wit the filter.  In this case, we
// only register a listener with the ring section filter
// because we don't care about the simple section filter
// at the moment.
SectionFilterListener myListener;
myListener = new MyListener();
ringFilter.addSectionFilterListener(myListener);

// We've now created all the section filters that we'll
// use, we can set some parameters.  Since we haven't
// attached to a transport stream yet, calling the
// startFiltering() method will simply set the parameters
// on the filter.  In this case, we set the ring section
// filter to match table ID 116 (the application
// information table) on PID 100.
ringFilter.startFiltering(null, 100, 116);

// Having set the parameters, we can attach the section
// filter group to a transport stream.  This will
// automatically start the filters once the group is
// attached
filterGroup.attach(ourTransportStream);

// At this point, our filtering is active and we just
// have to wait for some events.

Now lets take a look at the class that implements the listener for this section filter:

public class MyListener implements SectionFilterListener {

  // tells us where in the buffer we can find the next
  // section to read
  public int currentSectionIndex = 0;

  public void sectionFilterUpdate(
    SectionFilterEvent event) {

    // get the filter that the event was received from.
    // In this case, we assume it's a ring section
    // filter.
    RingSectionFilter filter;
    filter  = (RingSectionFilter) event.getSource();

    // Check that we've actually got a section.
    if (event instanceof SectionAvailableEvent) {

      // get the sections from the filter
      Section[] sections = filter.getSections();

      // now get the current section;
      Section currentSection;
      currentSection = sections[currentSectionIndex];

      // get the section data.  We could also choose to
      // get some of the section headers if we were more
      // interested in those.
      byte[] sectionData = currentSection.getData();

      // if we've got all the data we need from this
      // section, we free it so that this slot in the
      // buffer can be re-used and increment the index
      // for the current slot in the buffer
      currentSection.setEmpty();
      currentSectionIndex++;
      currentSectionIndex = currentSectionIndex % 5;

      // now we can do what we want to with the data from
      // the section without having to worry so much
      // about the buffer filling.
    }
  }
}

It’s important to remember that filtered sections should ideally be dealt with fairly quickly, so that the event handler is available to process more events. As sections are filtered, they will take up slots in the buffer even if your application hasn’t yet received the event.

An event handler that takes too much time to execute is likely to block notification of other section-related events, so it’s best for the event handler to simply extract the data from the section and then pass it to a separate thread for processing. This allows the event handler to return quickly, while still allowing the application to do a reasonable amount of processing on the data. Obviously, in the cases where only very simple processing is being done, such as the extraction of a single field from the section, this is not necessary.