raben.com

This is an example drupal module which provides a "RESTful" space weather service. It is used to provide data for the JavaFx Planetary Kp barchart.

If you are unfamiliar with the process of developing a drupal module,  you might want to first read the article "Creating modules - a tutorial:Drupal 6.x".

When the module is installed  (see file spaceweather.install) it creates one data table "spaceweather_kp_afwa_3hr". A planetary K indices consists of a date and time with an integer value. As provided by the NOAA space weather prediction center a K indices is in the range of 0 to 9 with missing values indicated as -1.  There is  slight complication though because final corrected K indices from the National Geophysical Data Center range in 28 steps from 0 to 9 with the fractional parts expressed in thirds of a unit.  These fractional values are published using a +/- notation. For example 3- is 3 minus 1/3 or 27 and 3+ is 3 plus 1/3 or 33.  To accomodate both in a single data table all that is necessary is to scale the K indices from Space Weather Prediction Center by 10.  They are then compatible with  the corrected indices received from National Geophysical Data Center.

The NOAA spaceweather prediction center publishes the K indices received from the Airforce Weather Agency (AFWA) every three hours at URL http://www.swpc.noaa.gov/ftpdir/indices/DGD.txt.  We need to retrieve the page from that URL,  parse the K indices from the page, and store them in the kp data table.

To retrieve the data periodically, the module implements the drupal hook_cron (see spaceweather.module) which is called by drupal cron.php.  For info on how this is set up see this drupal basic installation article on cron. Since cron may be called more often than every 3 hours the spaceweather_cron function checks the last time the K Indices were retrieved and calls the function spaceweather_get_kp_afwa_3hr when it has been longer than 3 hours since the last retrieval (lines 76 to 79).

/**
 * Implementation of hook_cron
 * Retrieve space weather data as required
 */
function spaceweather_cron() {
    // Get the last time kp data was updated
    $last_kp = variable_get('spaceweather_kp_afwa_3hr_time', 0);
    $age_kp = time() - $last_kp;

    // Retrieve kp data if its been longer than 3 hours
    if ($age_kp >= KP_AFWA_3HR_INTERVAL) {
        spaceweather_get_kp_afwa_3hr();
        variable_set('spaceweather_kp_afwa_3hr_time', time());
    }
}

The function spaceweather_get_kp_afwa_3hr retrieves the data from the Space Weather Prediction Center using the drupal function "drupal_http_request" (Line 87).  The return status is checked (Line 89 to 93) and then the php function "preg_match_all" (Line 96) scrapes the data from the text received. The function "preg_match_all" parses the text using the regular expression defined in variable KP_AFWA_3HR_PATTERN (Line 3 and 4) -- not shown here. This is a somewhat long but straight forward regular expression that identifies the patterns of numbers and spaces in the web page table of K Indices.  Then we loop through the array, picking off the date fields and 8 K Indice fields for each day.  A check is made to determine if the values are already in the data table (Lines 107 to 109). The values are then inserted or updated (Lines 120 to 126) into the data table.

/**
 * Retrieve kp data and insert into database
 */
function spaceweather_get_kp_afwa_3hr() {
    $kp_url = variable_get('spaceweather_kp_afwa_3hr_url', KP_AFWA_3HR_URL);
    $result = drupal_http_request($kp_url);

    if ($result->code != 200) {
        watchdog('spaceweather',"Error: spaceweather_get_kp from $kp_url" .
            " ret code=" . $result->code);
        return;
    }

    $html = $result->data;
    preg_match_all(KP_AFWA_3HR_PATTERN, $html, $matches, PREG_SET_ORDER);

    foreach ($matches AS $val) {
        $yrmoda = sprintf('%4d-%02d-%02d',$val[1],$val[2], $val[3]);
        $kparr = array($val[5], $val[6], $val[7],$val[8], $val[9], $val[10],
            $val[11], $val[12]);

        $hr = 0;

        foreach ($kparr AS $kp) {
            $date = "$yrmoda $hr";
            $result1 = db_query('SELECT date FROM {spaceweather_kp_afwa_3hr}'.
                " WHERE date='%s'", $date);
            $found = db_result($result1);

            // Note kp values are scaled by 10 to handle final +/- kp values
            // from NGDC. AFWA Kp values are always integer 1 to 9 only,
            // or -1 for missing value.
            if ($kp > 0) {
                $kpval = $kp * 10;
            } else {
                $kpval = $kp;
            }

            if (empty($found)) {
                db_query("INSERT INTO {spaceweather_kp_afwa_3hr} (date,kp) " .
                    "VALUES('%s',%d)", $date, $kpval);
            } else {
                db_query("UPDATE {spaceweather_kp_afwa_3hr} ".
                    "SET kp=%d WHERE date='%s'", $kpval, $date);
            }

            $hr += 3;
        }
    }

}

If the Kp bar graph applet calls a URL such as "http://localhost/swd/kp_afwa_3hr" the menu routes it the function "spaceweather_avail" which is defined in the file "spaceweather.public.inc". The function "spaceweather_avail" queries the data table for the date of the first K Indices and the last date that it is available.   The drupal function "json_encode" converts the  class object to JSON markup which is returned to the Planetary Kp bar graph applet.

When the bar graph applet calls "http://localhost/swd/kp_afwa_3hr/2010-04-24/4d" the menu defined in "spaceweather.module"  routes it to the "spaceweather_get" function. The values "kp_afwa_3hr", "2010-04-24" and "4d" are passed as function parameters "dataSet", "end", and "duration". The "dataSet" parameter specifies which data to retrieve. The "end" parameter specifies the end date. The "duration" parameter specifies the duration or time period to be retrieved prior to the end date.

Since the function may be publicly accessible, we don't want to use any parameters directly in the subsequent queries. For example the dataSet name is looked up and checked in lines 58 to 66. Duration is converted into an offset time value (Lines 79 to 102). The end date is parsed in spaceweather_date_string_to_array function (Lines 128 to 157) and the start and end dates are computed from the offset. The data is retrieved from the database (Lines 109 to 117) and added to a class structure.  The HTTP content type is set JSON in Line 120. The time series data class structure is then converted to JSON notation in Line 123.

**
 * Retrieve space weather data from the database
 * @param <type> $dataSet
 * @param <type> $end
 * @param <type> $duration 
 */
function spaceweather_get($dataSet='kp_afwa_3hr', $end=null, $duration='3d') {
    $SWD_TABLES = unserialize(SWD_TABLES);
    $SWD_INTERVALS = unserialize(SWD_INTERVALS);
    $tsd = new stdClass();
    $table = $SWD_TABLES[$dataSet];

    if (empty($table)) {
        drupal_set_header("Content-type: application/json");
        $tsd->message = t('dataSet is not recognized');
        $tsd->valid = false;
        print drupal_to_js($tsd);
        exit();
    }

    $tsd->valid = true;
    $tsd->dataset = $dataSet;
    $tsd->duration = $duration;

    if (empty($end)) {
        $end = gmdate('Y-m-d H:i');
    }

    $tsd->end = $end;
    $tsd->interval = $SWD_INTERVALS[$dataSet];
    $date = spaceweather_date_string_to_array($end);
    preg_match('/(\d+)([dhms])/', $duration, $dur);

    if (count($dur) != 3) {
         drupal_set_header("Content-type: application/json");
        $tsd->message = t('Duration is not valid');
        $tsd->valid = false;
        print drupal_to_js($tsd);
        exit();
    }

    switch($dur[2]) {
        case 'd' : $offset = $dur[1] * 86400; break;
        case 'h' : $offset = $dur[1] * 3600; break;
        case 'm' : $offset = $dur[1] * 60; break;
        case 's' : $offset = $dur[1]; break;
        default:
            $tsd->message = t('Interval not recognized; set to 3 days (3d)');
            $offset = 86400 * 3; // 3 day by default
    }

    // Limit request to 10 days max
    if ($offset > 864000) {
        $offset = 864000;
    }

    $time = gmmktime(0, 0, 0, $date[1], $date[2], $date[0]);
    $end = gmdate('Y-m-d H:i:s', $time + 86399);
    $start = gmdate('Y-m-d H:i:s', ($time + 86400) - $offset);
    $tsd->start = $start;
    $tsd->vals = array();
    $sql = sprintf("SELECT * FROM $table" .
        " WHERE date BETWEEN '%s' AND '%s'", $start, $end);

    $result = db_query("SELECT * FROM $table" .
        " WHERE date BETWEEN '%s' AND '%s'", $start, $end);
    while ($row = db_fetch_array($result)) {
        $tsd->vals[] = array('date' => $row['date'], 'name' => 'kp',
            'val' => $row['kp']);
    }


    drupal_set_header("Content-type: application/json");
    drupal_set_header("Pragma: no-cache");
    header("Expires: 0");
    print json_encode($tsd);
    exit();

}

Download file: 

spaceweather.tar.gz