OpenNMS API 1.2.9

## org.opennms.netmgt.rrd
Class QueuingRrdStrategy

    java.lang.Object
      org.opennms.netmgt.rrd.QueuingRrdStrategy

All Implemented Interfaces:
  RrdStrategy, java.lang.Runnable

class QueuingRrdStrategy
extends java.lang.Object
implements RrdStrategy, java.lang.Runnable

Provides queueing implementation of RrdStrategy. In order to provide a more scalable collector. We created a queuing RrdStrategy that enabled the system to amortize the high cost of opening an round robin database across multiple updates. This RrdStrategy implementation enqueues the create and update operations on a per file basis and maintains a set of threads that process enqueued work file by file. If the I/O system can keep up with the collection threads while performing only a single update per file then eventually all the data is processed and the threads sleep until there is more work to do. If the I/O system is initially slower than than the collection threads then work will enqueue here and the write threads will get behind. As this happens each file will eventually have more than a single update enqueued and therefore the number of updates pushed thru the system will increase because more then one will be output per 'open' Eventually, the I/O system and the collection system will balance out. When this happens all data will be collected but will not be output to the rrd files until the next time the file is processed by the write threads. As another performance improving strategy. The queue distinguishes between files with signficant vs insignifact updates. Files with only insignificant updates are put at the lowest priority and are only written when the highest priority updates have been written This implementation delegates all the actual writing to another RrdStrategy implementation. System properites effecting the operation: org.opennms.rrd.queuing.writethreads: (default 2) The number of rrd write threads that process the queue org.opennms.rrd.queuing.queueCreates: (default true) indicates whether rrd file creates should be queued or processed synchronously org.opennms.rrd.queuing.maxInsigUpdateSeconds: (default 0) the number of seconds over which all files with significant updates only should be promoted onto the significant less. This is to ensure they don't stay unprocessed forever. Zero means not promotion. org.opennms.rrd.queuing.modulus: (default 10000) the number of updates the get enqueued between statistics output org.opennms.rrd.queuing.category: (default "UNCATEGORIZED") the log category to place the statistics output in TODO: Promote files when ZeroUpdate operations can't be merged. This may be a collection miss which we want to push thru. It should also help with memory. TODO: Set an upper bound on enqueued operations TODO: Provide an event that will write data for a particular file... Say right before we try to graph it.

Nested Class Summary
`class` `QueuingRrdStrategy.CreateOperation` This class represents an operation to create an rrd file
`(package private) static class` `QueuingRrdStrategy.Operation` This is the base class for an enqueueable operation
`class` `QueuingRrdStrategy.UpdateOperation` Represents an update to a rrd file.
`class` `QueuingRrdStrategy.ZeroUpdateOperation` Represents an update whose value is 0.

Field Summary
`(package private) static int` `CREATE`
`(package private) long` `createsCompleted`
`(package private) long` `dequeuedItems`
`(package private) long` `dequeuedOperations`
`(package private) long` `enqueuedOperations`
`(package private) long` `errors`
`(package private) java.util.Map` `fileAssignments`
`(package private) java.util.LinkedList` `filesWithInsignificantWork`
`(package private) java.util.LinkedList` `filesWithSignificantWork`
`private static long` `INSIG_HIGH_WATER_MARK`
`(package private) long` `lastDequeued`
`(package private) long` `lastDequeuedItems`
`(package private) long` `lastEnqueued`
`(package private) long` `lastLap`
`(package private) long` `lastOpsPending`
`(package private) long` `lastSignificantCompleted`
`(package private) long` `lastSignificantDequeued`
`(package private) long` `lastSignificantEnqueued`
`(package private) long` `lastStatsTime`
`private static java.lang.String` `LOG4J_CATEGORY`
`(package private) RrdStrategy` `m_delegate`
`private static long` `MAX_INSIG_UPDATE_SECONDS`
`private static long` `MODULUS`
`(package private) java.util.Map` `pendingFileOperations`
`private static boolean` `PRIORITIZE_SIGS`
`(package private) long` `promotionCount`
`private static boolean` `QUEUE_CREATES`
`private static long` `QUEUE_HIGH_WATER_MARK`
`(package private) java.util.Set` `reservedFiles`
`private static long` `SIG_HIGH_WATER_MARK`
`(package private) long` `significantOpsCompleted`
`(package private) long` `significantOpsDequeued`
`(package private) long` `significantOpsEnqueued`
`(package private) int` `threadsRunning`
`(package private) long` `totalOperationsPending`
`(package private) static int` `UPDATE`
`(package private) long` `updatesCompleted`
`(package private) long` `updateStart`
`private static long` `WRITE_THREAD_EXIT_DELAY`
`private static long` `WRITE_THREAD_SLEEP_TIME`
`(package private) static int` `WRITE_THREADS`

Constructor Summary
`QueuingRrdStrategy(RrdStrategy delegate)`

Method Summary
`void` `addOperation(QueuingRrdStrategy.Operation op)` Add an operation to the queue.
`void` `closeFile(java.lang.Object rrd)` This closes the supplied round robin database
`private void` `completeAssignment()` Record that fact that the current thread has finished process operations for its current assignement
`java.lang.Object` `createDefinition(java.lang.String creator, java.lang.String directory, java.lang.String dsName, int step, java.lang.String dsType, int dsHeartbeat, java.lang.String dsMin, java.lang.String dsMax, java.util.List rraList)` Create a round robin database definition from the supplied parameters.
`void` `createFile(java.lang.Object op)` Creates the round robin database defined by the supplied definition.
`java.io.InputStream` `createGraph(java.lang.String command, java.io.File workDir)` Creates an InputStream representing the bytes of a graph created from round robin data.
`void` `ensureThreadsStarted()` Ensure that we have threads started to process the queue.
`java.lang.Double` `fetchLastValue(java.lang.String rrdFile, int interval)` Fetches the last value from the round robin database with the given name.
`java.lang.Double` `fetchLastValueInRange(java.lang.String rrdFile, int interval, int range)` Fetches the last value from the round robin database with the given name within a time range.
`java.lang.String` `getLapTime()`
`java.util.LinkedList` `getNext()` Get the operations for the next file that should be worked on.
`java.lang.String` `getStats()` Print queue statistics.
`void` `graphicsInitialize()` This Initializes the graphics subsystem only.
`private boolean` `hasOnlyInsignificant(java.util.LinkedList pendingOps)` Return true if and only if all the operations in the list are insignificant
`void` `initialize()` Initialize the appropriate round robin system
`private boolean` `inSigQueueIsFull()`
`private org.apache.log4j.Category` `log()`
`private void` `log(java.lang.String msg)`
`QueuingRrdStrategy.Operation` `makeCreateOperation(java.lang.String fileName, java.lang.Object rrdDef)`
`QueuingRrdStrategy.Operation` `makeUpdateOperation(java.lang.String fileName, java.lang.String update)`
`java.lang.Object` `openFile(java.lang.String fileName)` Opens the round robin database with the supplied name.
`(package private) void` `printLapTime(java.lang.String message)`
`void` `printStats()`
`private void` `processClose(java.lang.Object rrd)` close the rrd file
`private void` `processPendingOperations()` Actually process the operations by calling the underlying delegate strategy
`private void` `promoteAgedFiles()` Ensure that files with insignificant changes are getting promoted if necessary
`private boolean` `queueIsFull()`
`void` `run()`
`private java.lang.String` `selectNewAssignment()` Return the name of the next file with available work
`private boolean` `sigQueueIsFull()`
`private void` `storeAssignment(QueuingRrdStrategy.Operation op)` We need to track which files are being processed by which threads so that we don't try to process updates for the same file on more than one thread.
`private java.util.LinkedList` `takeAssignment(java.lang.String newAssignment)` register the file that the currentThread is be working on.
`void` `updateFile(java.lang.Object rrdFile, java.lang.String data)` Updates the supplied round robin database with the given timestamp:value point

Methods inherited from class java.lang.Object
`clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

Field Detail

### m_delegate

    RrdStrategy m_delegate

### UPDATE

    static final int UPDATE

See Also:
  Constant Field Values

### CREATE

    static final int CREATE

See Also:
  Constant Field Values

### WRITE_THREADS

    static final int WRITE_THREADS

### QUEUE_CREATES

    private static final boolean QUEUE_CREATES

### PRIORITIZE_SIGS

    private static final boolean PRIORITIZE_SIGS

### INSIG_HIGH_WATER_MARK

    private static final long INSIG_HIGH_WATER_MARK

### SIG_HIGH_WATER_MARK

    private static final long SIG_HIGH_WATER_MARK

### QUEUE_HIGH_WATER_MARK

    private static final long QUEUE_HIGH_WATER_MARK

### MODULUS

    private static final long MODULUS

### LOG4J_CATEGORY

    private static final java.lang.String LOG4J_CATEGORY

### MAX_INSIG_UPDATE_SECONDS

    private static final long MAX_INSIG_UPDATE_SECONDS

### WRITE_THREAD_SLEEP_TIME

    private static final long WRITE_THREAD_SLEEP_TIME

### WRITE_THREAD_EXIT_DELAY

    private static final long WRITE_THREAD_EXIT_DELAY

### filesWithSignificantWork

    java.util.LinkedList filesWithSignificantWork

### filesWithInsignificantWork

    java.util.LinkedList filesWithInsignificantWork

### pendingFileOperations

    java.util.Map pendingFileOperations

### fileAssignments

    java.util.Map fileAssignments

### reservedFiles

    java.util.Set reservedFiles

### totalOperationsPending

    long totalOperationsPending

### enqueuedOperations

    long enqueuedOperations

### dequeuedOperations

    long dequeuedOperations

### significantOpsEnqueued

    long significantOpsEnqueued

### significantOpsDequeued

    long significantOpsDequeued

### significantOpsCompleted

    long significantOpsCompleted

### dequeuedItems

    long dequeuedItems

### createsCompleted

    long createsCompleted

### updatesCompleted

    long updatesCompleted

### errors

    long errors

### threadsRunning

    int threadsRunning

### updateStart

    long updateStart

### promotionCount

    long promotionCount

### lastLap

    long lastLap

### lastStatsTime

    long lastStatsTime

### lastEnqueued

    long lastEnqueued

### lastDequeued

    long lastDequeued

### lastSignificantEnqueued

    long lastSignificantEnqueued

### lastSignificantDequeued

    long lastSignificantDequeued

### lastSignificantCompleted

    long lastSignificantCompleted

### lastDequeuedItems

    long lastDequeuedItems

### lastOpsPending

    long lastOpsPending

Constructor Detail

### QueuingRrdStrategy

    public QueuingRrdStrategy(RrdStrategy delegate)

Method Detail

### makeCreateOperation

    public QueuingRrdStrategy.Operation makeCreateOperation(java.lang.String fileName,
                                                            java.lang.Object rrdDef)

### makeUpdateOperation

    public QueuingRrdStrategy.Operation makeUpdateOperation(java.lang.String fileName,
                                                            java.lang.String update)

### addOperation

    public void addOperation(QueuingRrdStrategy.Operation op)

  Add an operation to the queue.

### log

    private org.apache.log4j.Category log()

### queueIsFull

    private boolean queueIsFull()

### sigQueueIsFull

    private boolean sigQueueIsFull()

### inSigQueueIsFull

    private boolean inSigQueueIsFull()

### ensureThreadsStarted

    public void ensureThreadsStarted()

  Ensure that we have threads started to process the queue.

### getNext

    public java.util.LinkedList getNext()

  Get the operations for the next file that should be worked on.

  Returns:
    a linkedList of oeprations to be processed all for the same file.

### storeAssignment

    private void storeAssignment(QueuingRrdStrategy.Operation op)

  We need to track which files are being processed by which threads so that we don't try to process updates for the same file on more than one thread.

### promoteAgedFiles

    private void promoteAgedFiles()

  Ensure that files with insignificant changes are getting promoted if necessary

### hasOnlyInsignificant

    private boolean hasOnlyInsignificant(java.util.LinkedList pendingOps)

  Return true if and only if all the operations in the list are insignificant

### takeAssignment

    private java.util.LinkedList takeAssignment(java.lang.String newAssignment)

  register the file that the currentThread is be working on. This enables us to ensure that another thread doesn't try to work on operations for that file.

### selectNewAssignment

    private java.lang.String selectNewAssignment()

  Return the name of the next file with available work

### completeAssignment

    private void completeAssignment()

  Record that fact that the current thread has finished process operations for its current assignement

### closeFile

    public void closeFile(java.lang.Object rrd)
                   throws java.lang.Exception

  Description copied from interface: `RrdStrategy`
  This closes the supplied round robin database

  Specified by:
    `closeFile` in interface `RrdStrategy`
  Parameters:
    `rrd` - an rrd object created using openFile
  Throws:
    `java.lang.Exception` - if an error occurs closing the file

### createDefinition

    public java.lang.Object createDefinition(java.lang.String creator,
                                             java.lang.String directory,
                                             java.lang.String dsName,
                                             int step,
                                             java.lang.String dsType,
                                             int dsHeartbeat,
                                             java.lang.String dsMin,
                                             java.lang.String dsMax,
                                             java.util.List rraList)
                                      throws java.lang.Exception

  Description copied from interface: `RrdStrategy`
  Create a round robin database definition from the supplied parameters. This definition is used in the createFile call to create the actual file.

  Specified by:
    `createDefinition` in interface `RrdStrategy`
  Parameters:
    `creator` - - A string representing who is creating this file for use in log msgs
    `directory` - - The directory to create the file in
    `dsName` - - The datasource name for use in the round robin database
    `step` - - the step for the database
    `dsType` - - the type for the datasource
    `dsHeartbeat` - - the heartbeat for the datasouce
    `dsMin` - - the minimum allowable value for the datasource
    `dsMax` - - the maximum allowable value for the datasouce
    `rraList` - - a List of the round robin archives to create in the database
  Returns:
    an object representing the definition of an round robin database
  Throws:
    `java.lang.Exception` - If an error occurs while creating the definition

### createFile

    public void createFile(java.lang.Object op)
                    throws java.lang.Exception

  Description copied from interface: `RrdStrategy`
  Creates the round robin database defined by the supplied definition.

  Specified by:
    `createFile` in interface `RrdStrategy`
  Parameters:
    `op` - an round robin database definition created using the createDefinition call.
  Throws:
    `java.lang.Exception` - if an error occurs create the file

### initialize

    public void initialize()
                    throws java.lang.Exception

  Description copied from interface: `RrdStrategy`
  Initialize the appropriate round robin system

  Specified by:
    `initialize` in interface `RrdStrategy`
  Throws:
    `java.lang.Exception` - if an Error occurs

### graphicsInitialize

    public void graphicsInitialize()
                            throws java.lang.Exception

  Description copied from interface: `RrdStrategy`
  This Initializes the graphics subsystem only. This is used by the web application to avoid the need for the JNI infrastructure in the webapp.

  Specified by:
    `graphicsInitialize` in interface `RrdStrategy`
  Throws:
    `java.lang.Exception`

### openFile

    public java.lang.Object openFile(java.lang.String fileName)
                              throws java.lang.Exception

  Description copied from interface: `RrdStrategy`
  Opens the round robin database with the supplied name. It is assumed the name refers to a round robin database appropriate for this strategy implementation

  Specified by:
    `openFile` in interface `RrdStrategy`
  Parameters:
    `fileName` - the name of the associated rrd file
  Returns:
    an open rrd reference that can by used in calls to updateFile and closeFile
  Throws:
    `java.lang.Exception` - if an error occurs opening the file

### updateFile

    public void updateFile(java.lang.Object rrdFile,
                           java.lang.String data)
                    throws java.lang.Exception

  Description copied from interface: `RrdStrategy`
  Updates the supplied round robin database with the given timestamp:value point

  Specified by:
    `updateFile` in interface `RrdStrategy`
  Parameters:
    `rrdFile` - an rrd object created using openFile
    `data` - a string of the form <timestamp>: <datavalue>
  Throws:
    `java.lang.Exception` - if an error occurs updating the file

### fetchLastValue

    public java.lang.Double fetchLastValue(java.lang.String rrdFile,
                                           int interval)
                                    throws java.lang.NumberFormatException,
                                           RrdException

  Description copied from interface: `RrdStrategy`
  Fetches the last value from the round robin database with the given name. The interval passed in should be the interval associated with the round robin database.

  Specified by:
    `fetchLastValue` in interface `RrdStrategy`
  Parameters:
    `rrdFile` - a name the represents a round robin database
    `interval` - a step interval of the round robin database
  Returns:
    The last value as a Double (if the last value didn't exist returns a Double.NaN)
  Throws:
    `RrdException`
    `java.lang.NumberFormatException`

### fetchLastValueInRange

    public java.lang.Double fetchLastValueInRange(java.lang.String rrdFile,
                                                  int interval,
                                                  int range)
                                           throws java.lang.NumberFormatException,
                                                  RrdException

  Description copied from interface: `RrdStrategy`
  Fetches the last value from the round robin database with the given name within a time range. The interval passed in should be the interval associated with the round robin database. The range should be the amount of "lag" acceptable for an update to be considered valid. Range must be a multiple of the RRD interval.

  Specified by:
    `fetchLastValueInRange` in interface `RrdStrategy`
  Parameters:
    `rrdFile` - a name the represents a round robin database
    `interval` - a step interval of the round robin database
    `range` - an acceptable range for which the last value will be returned
  Returns:
    The last value as a Double (if the last value didn't exist returns a Double.NaN)
  Throws:
    `RrdException`
    `java.lang.NumberFormatException`

### createGraph

    public java.io.InputStream createGraph(java.lang.String command,
                                           java.io.File workDir)
                                    throws java.io.IOException,
                                           RrdException

  Description copied from interface: `RrdStrategy`
  Creates an InputStream representing the bytes of a graph created from round robin data. It accepts an rrdtool graph command. The underlying implementation converts this command to a format appropriate for it .

  Specified by:
    `createGraph` in interface `RrdStrategy`
  Parameters:
    `command` - the command needed to create the graph
    `workDir` - the directory that all referenced files are relative to
  Returns:
    an input stream representing the bytes of a graph image as a PNG file
  Throws:
    `java.io.IOException` - if an IOError occurs
    `RrdException` - if an RRD error occurs

### run

    public void run()

  Specified by:
    `run` in interface `java.lang.Runnable`

### processPendingOperations

    private void processPendingOperations()

  Actually process the operations by calling the underlying delegate strategy

### processClose

    private void processClose(java.lang.Object rrd)

  close the rrd file

### getStats

    public java.lang.String getStats()

  Print queue statistics.

  Specified by:
    `getStats` in interface `RrdStrategy`
  Returns:
    a non-null string representing any staticstics to be included in the logs

### printStats

    public void printStats()

### printLapTime

    void printLapTime(java.lang.String message)

### log

    private void log(java.lang.String msg)

  Parameters:
    `msg` -

### getLapTime

    public java.lang.String getLapTime()

OpenNMS API 1.2.9

Generated by ranger on May 19 2007 2335.