Acumen Development

The modern concept of ‘web services’ has always existed in one form or another. Originally the protocols were binary and incredibly complex to integrate, but times have changed a lot. However, there are still large swathes of service providers who provide web-based APIs, but only support primitive versions of these- relying on ‘file drops’ and simple callbacks. Worse still, these providers are particularly resistant to changing their ways. This can be a serious bar of entry for more modern development companies who are used to a SOAP/REST based system, stopping them climb the supplier chain.

This short example shows how you can use PHP to integrate with a supplier who insists on using a file drop mechanism – in this case, to get your date-stampped CSV files, prefixed “file-” wrapped up in ZIP file using the format user-DMY, on an FTP site of their choice!

$host='www.example.com';
$user='myuser';
$pass='mypass';
$date=date('dmy');

// where to store file
$store='/tmp/files.zip';

// get file
$ch = curl_init('ftp://{$user}:{$pass}@{$host}/{$user}-{$date}.zip');
curl_setopt($ch,CURLOPT_RETURNTRANSFER,1);
$file=curl_exec($ch);
file_put_contents($store,$file);

// output ids of all entries
$entries = getEntriesFromCSV(getEntryFile($store) );
foreach($entries as $entry){
    print '{$entry['id']}\n';
}

/*
* @desc   Get CSV number file from local ZIP file
* @author Leo Brown
* @param  $file String Local file to open
*/
function getEntryFile($file)
{
    if ($file &
    &
    is_resource($zip = zip_open($file))) {
        while ($entry=zip_read($zip)) {
            if ('file' == substr(zip_entry_name($entry),0,4)) {
                return zip_entry_read($entry, zip_entry_filesize($entry
                )
                );
            }
        }
    }
}

/*
* @desc   Get provider entries from CSV formatted string
* @author Leo Brown
* @param  $file String Allocated entries as 'entry,t1,t2,t3\n'
*/
function getEntriesFromCSV($file)
{
    $lines=explode('\n',$file);
    $entries=array();
    foreach($lines as $line){
        $fields=explode(',',trim($line));
        if (is_numeric($fields[0])) {
            $entry=array('id'=>$fields[0],
            'data1'=>@$fields[1],
            'data2'=>@$fields[2],
            'data3'=>@$fields[3]
            );
            $entry[$fields[0]]=array_filter($entry);
        }
    }
    return $entries;
}
?>

There are numerous applications for integration of inbound email – the most obvious being ticketing systems, and robot systems (auto/interactive responders).

This is really easily achievable on UNIX systems using the .forward (dot forward) file and your scripting language of choice. If you wanted to pipe all of root’s mail into a script, you would create a .forward file in that user’s home directory, consisting of a single line, starting with a pipe, followed by the full process path.

|/root/process.php

Then you need to provide a destination processor for your email. In this case we have used a UNIX-style file processor at the start by using the hashbang syntax:

#!/usr/local/bin/php
<?php

// read from stdin
$fp = fopen('php://stdin', 'r');
while(!feof($fp)) {
$lines[]=fgets($fp, 4096);
}

[... email processing code...]
?>

You can then use an optional MIME decoder such as Pear mimeDecode to break up the email into the relevant sections – for instance, the from/to fields and plain and/or HTML body text.

Choosing a File Transfer Technology

There are dozens of approaches to the actual process of transferring software data between systems. Most of these are dependent on the operating systems in use, but there are also important speed and security considerations for each approach. Usually when transferring data across the web, security (specifically interceptability) will be of the utmost importance, so we’ll concentrate on transfer methods that are fairly secure, such as:
 

• SCP (Session Control Protocol)
• SFTP (FTP over SSH)
• HTTPS POST
• Encrypted files by FTP
• Encrypted files by MIME Email

Software and Operating System Considerations

Both SCP and SFTP are only supported natively on UNIX systems. Web-based transfers rely on the destination system running a webserver.

Network / Firewall Considerations

The first thing you may need to consider in creating a sustainable process is what’s feasible given network conditions. If you’re exchanging data between two unfirewalled servers in the same datacentre, this won’t be an issue – but this is a rare case. SSH and it’s subsets (SFTP, SCP) use only a single port: TCP 22, so you shouldn’t expect any problems there. FTP however (c. 1971!) is an oblique, firewall-hating protocol. If only one system is to be firewalled, there is a (fairly) straightforward solution – FTP passive mode. This puts the open port requirements back to the client host rather than the server.

The remaining protocols, HTTPS (TCP 443), and Email (SMTP TCP 25) are fairly firewall-friendly and shouldnt cause any problems. However, due to the large-scale uptake of HTTP and SMTP for ‘general use’ on the internet, you may find yourself dealing with a never ending set of test cases, due to the many proxying and anti-spam measures out there.

Choosing a Data Interchange Method

In the case that you are transferring one-dimensional data (such as a series of JPEG images to be printed, with no data beyond their EXIF tags), then you won’t need to prepare your data. For something a bit more complex, such as customer records to be imported into a third party database, you’ll want to choose a robust interchange method.

Plain Text Data

In many contexts, it may be ­tempting to use a CSV format for data interchange, simply because it’s human readable, and can be loaded into spreadsheet software like Excel by managers or supervisors who may want to see the data in “raw form” from time to time. If you have a matching data sink and source, such as Excel both generating and reading your end data, then this approach can be sufficient.

In reality, CSV can be a nightmare to implement. When it comes to exporting, the first hurdle is often that while most plain text encoding will “escape” fields with a backslash, CSV escapes using the double quote character.

Mixed / Binary Data

For more complex (non-ASCII, or binary) data, your life can be made easier using a markup language such as XML or a specific subset relating to the type of data you’re transferring.

Other Options

If exporting or mirroring data with a flexible form, your database layer may be able to provide a resillient export format saving you a lot of extra development. For instance, the MySQL utility mysqldump supports XML output with the -x option.