Acumen Development

If there’s two bits of software I love, Wordpress and Subversion would be right up there in terms of making my life simpler, and making publishing easier. While it’s tempting to counter statements like these with a long list of problems, I’ll leave that as it is – today I want to convey a few things that helped me use the two pieces of software together.

First, the context of using Wordpress and Subversion jointly is typically in making sure that customer websites are fully versioned so that we can be accountable and transparent, and equally importantly so that we can roll back in a disaster scenario.

Second, some background on our approach. We believe that when using Open Source software for customers, it’s important to strike a balance between cutting edge features and stability. We typically accomplish this by starting the project at the head revision of the software we’re using – in this case, Wordpress.

svn co http://core.svn.wordpress.org/trunk www

Now we have a freshly checked out copy of Wordpress, and we’ll want to keep it this way. If you don’t feel comfortable with leaving a repository in a certain state there are a few options here – the most obvious would be for you to maintain your own copy of the Wordpress source tree (which can be useful in other ways too, for instance if WP’s subversion servers were to go down.) This can be accomplished easily using `svnsync`.

Assuming that you’re happy with your checked-out Wordpress, we now want to move all the User data to one place, so that we can version it separately to the Wordpress trunk. Typically, this is just the `wp-content` folder, but you may also want to version wp-config.php, .htaccess, and any other special files you’re using in the project webroot.

Your final decision on the path structure really depends on the scenario- you may want to separate out the theme directory from other ‘user content’ directories if you’re planning to open source your theme or host it elsewhere. In this case, you’d have a subversion repository containing the folders within wp-content, with a `themes` folder which will load your versioned theme(s) as an svn:externals entry.

Now, a couple of tricks – first, your uploads folder may want to be versioned. If not, you’ll want to svn:ignore it. If you do version it, you’ll probably want to svn:ignore some of it. For instance, WP2.5 onwards supports image resizing, and it uses the uploads directory also as a cache store for these thumbnails. We can ignore these thumbnails by setting the svn:ignore property on the uploads directory as follows:

[root@www uploads]# svn -R  propset svn:ignore "*[0-9]x[1-9]*"; .
property 'svn:ignore' set (recursively) on '.'

Note that if your uploads folder is structured by date, these ignore settings will not persist in new directories, so I’d recommend you do not use a structured uploads folder if you are versioning it (the option titled ” Organize my uploads into month- and year-based folders”).

Subversion is a great piece of version control software, and has been at the core of all our development projects for many years. Sadly, there are a few weaknesses. Key among them is lack of obliterate support, but one that’s stung us a few times now is corrupt SVN databases.

Here’s a quick runthrough of the solution I developed to the common ‘bad transaction’ issue, that causes your repository to stop checking out/modifying certain files (typically blobs).

1. Use svnadmin verify to find your bad revision (in this example, 195)

svn verify myrepo
* Verified revision 0.
...
* Verified revision 194.
svnadmin: Malformed representation header

2. Dump the first working section of the repository to a dumpfile, then dump the second section as an incremental dump onto the end of this dumpfile

svnadmin dump myrepo -r0:194 > my.dmp
svnadmin dump myrepo --incremental -r196:HEAD >> my.dmp

3. Try to rebuild this repository by creating a new repository, and using svnadmin load to read in the newly created dumpfile.

svnadmin create newrepo
cat my_filtered.dmp | svnadmin load newrepo

4. It’s likely that paths can not be rebuilt, and you will see errors describing this. Use svndumpfilter to exclude the problematic paths, creating a new dumpfile for each exclusion, and repeating from step 3.

cat my.dmp | svndumpfilter exclude "My/Bad/Path" > my_filtered.dmp
Excluding prefixes:
   '/My/Bad/Path'

Revision 0 committed as 0. (etc)

5. Replace the rebuilt repository’s UUID with that of the old repository, by using svnlook uuid and svnadmin setuuid.

svnadmin setuuid newrepo `svnlook uuid myrepo`

6. Use svn switch to attempt to move a checked out copy to your new repository.

svn --relocate switch http://host/myrepo http://host/newrepo .

7. If all goes to plan, replace your corrupt repository with your rebuild, and svn switch any working copies back to their original path. Then copy back in the data that you lost in the exclusion process, commit and get on with your work.

There you go, loads of messing around to get back to square one. Once that’s done, stop using FSFS as your SVN backend, as it’s much more buggy!

The modern concept of ‘web services’ has always existed in one form or another. Originally the protocols were binary and incredibly complex to integrate, but times have changed a lot. However, there are still large swathes of service providers who provide web-based APIs, but only support primitive versions of these- relying on ‘file drops’ and simple callbacks. Worse still, these providers are particularly resistant to changing their ways. This can be a serious bar of entry for more modern development companies who are used to a SOAP/REST based system, stopping them climb the supplier chain.

This short example shows how you can use PHP to integrate with a supplier who insists on using a file drop mechanism – in this case, to get your date-stampped CSV files, prefixed “file-” wrapped up in ZIP file using the format user-DMY, on an FTP site of their choice!

$host='www.example.com';
$user='myuser';
$pass='mypass';
$date=date('dmy');

// where to store file
$store='/tmp/files.zip';

// get file
$ch = curl_init('ftp://{$user}:{$pass}@{$host}/{$user}-{$date}.zip');
curl_setopt($ch,CURLOPT_RETURNTRANSFER,1);
$file=curl_exec($ch);
file_put_contents($store,$file);

// output ids of all entries
$entries = getEntriesFromCSV(getEntryFile($store) );
foreach($entries as $entry){
    print '{$entry['id']}\n';
}

/*
* @desc   Get CSV number file from local ZIP file
* @author Leo Brown
* @param  $file String Local file to open
*/
function getEntryFile($file)
{
    if ($file &
    &
    is_resource($zip = zip_open($file))) {
        while ($entry=zip_read($zip)) {
            if ('file' == substr(zip_entry_name($entry),0,4)) {
                return zip_entry_read($entry, zip_entry_filesize($entry
                )
                );
            }
        }
    }
}

/*
* @desc   Get provider entries from CSV formatted string
* @author Leo Brown
* @param  $file String Allocated entries as 'entry,t1,t2,t3\n'
*/
function getEntriesFromCSV($file)
{
    $lines=explode('\n',$file);
    $entries=array();
    foreach($lines as $line){
        $fields=explode(',',trim($line));
        if (is_numeric($fields[0])) {
            $entry=array('id'=>$fields[0],
            'data1'=>@$fields[1],
            'data2'=>@$fields[2],
            'data3'=>@$fields[3]
            );
            $entry[$fields[0]]=array_filter($entry);
        }
    }
    return $entries;
}
?>

Today Acumen launched the new Peace Direct website.

Peace Direct’s cause is grassroots peacebuilding initiatives, and is the sister site of Acumen’s earlier project Insight on Conflict.

The site was built on Wordpress and utilises some of the latest features to create a great-looking, dynamic and easily maintainable site. You can access the site at the following link:

http://www.peacedirect.org

Acumen today launched the new website for the Centre for Investigative Journalism, a training and research charity committed to education and reporting based in London.

The site is built in WordPress, the open-source CMS, and is produced from designs provided by Typographics .

Visit the TCIJ site for more info: http://www.tcij.org.

Today Acumen launched a new service allowing Touchnote users to generate cards on the basis of their Facebook photos.

Try out the new app at http://www.touchnote.com/facebook

Article Cuttings:

• Touchnote launches API, integrates with Facebook photos
• Touchnote provides direct access to Facebook albums

Despite great Evangelism about the breakthroughs of Web2.0, the design limitations of HTML still live on – a total lack of CSS3 uptake, and a highly restrictive set of web fonts – barely three serifs and sans serifs to match. While possibilities with content and navigation have been opened up, design limitations of standard web pages are leaving DHTML standing for Dilapidated HTML, and has driven so many style-conscious sites to be poorly implemented in flash – which due to a lack of complex positioning tools can be very ignorant of viewport size. The birth of sIFR was a breakthrough crossover between Flash and HTML, but this still comes with some restrictions and risks that many companies find hard to justify on their sites.

At Acumen, we’ve never wanted to restrain designers too much by forcing them to use web fonts, but then we also take on a responsibility to offer our customers sites that they can fully control. Matting down a title font for each page makes it hard for webmasters without a design background to make the changes they need, and so we’ve used a number of tools along the way to bring that control back to the maintainers.

Key among these is a system to render non-webfonts on the fly, determined by user-generated content. You can see this in use on the Acumen site, for instance the use of Bank Gothic on the title of this page. This is accomplished using a WordPress plugin we’ve written to replace specific page tags and CSS selectors – and it works fairly seemlessly, requiring only Javascript and PHP to run.

Anyway, by request, we’ve decided to release the plugin for general use – enjoy!

Acumen are proud to report the release of the Varsani Richardson website. V+R engaged Acumen to deliver their exact typography requirements to the web, which Acumen accomplished with a range of technologies detailed in previous posts.

You can view the new V+R website at www.varsanirichardsonphoto.com

There are numerous applications for integration of inbound email – the most obvious being ticketing systems, and robot systems (auto/interactive responders).

This is really easily achievable on UNIX systems using the .forward (dot forward) file and your scripting language of choice. If you wanted to pipe all of root’s mail into a script, you would create a .forward file in that user’s home directory, consisting of a single line, starting with a pipe, followed by the full process path.

|/root/process.php

Then you need to provide a destination processor for your email. In this case we have used a UNIX-style file processor at the start by using the hashbang syntax:

#!/usr/local/bin/php
<?php

// read from stdin
$fp = fopen('php://stdin', 'r');
while(!feof($fp)) {
$lines[]=fgets($fp, 4096);
}

[... email processing code...]
?>

You can then use an optional MIME decoder such as Pear mimeDecode to break up the email into the relevant sections – for instance, the from/to fields and plain and/or HTML body text.

While in production applications we all favour use of an API, there are a lot of situations, such as in ‘Mashups’ (I love how that term has been reappropriated from Jungle music) where you need to do some page scraping.

It’s occurred to me how these very easy techniques seem inaccessible to many people, so I thought I’d post a few bits and bobs about some basic scraping methods.

Here’s a bit of code I wrote to use PHP’s DOMDocument class to treat a HTML page as XML and fetch, in this case, the incredibly useful current world population… fantastic!

<?php
 // where to find population data...
 $location['url']='http://www.census.gov/ipc/www/popclockworld.html';
 $location['id']='worldnumber';

 // initialise a new document and prepare the data
 $d=new DOMDocument();
 $file = file_get_contents($location['url']);

 // get and print current world population
 $d->loadHTML($file);
 $e=$d->getElementById($location['id']);
 print $e->nodeValue;
?>

Sample output: 6,738,610,278