Acumen Development

Subversion is a great piece of version control software, and has been at the core of all our development projects for many years. Sadly, there are a few weaknesses. Key among them is lack of obliterate support, but one that’s stung us a few times now is corrupt SVN databases.

Here’s a quick runthrough of the solution I developed to the common ‘bad transaction’ issue, that causes your repository to stop checking out/modifying certain files (typically blobs).

1. Use svnadmin verify to find your bad revision (in this example, 195)

svn verify myrepo
* Verified revision 0.
...
* Verified revision 194.
svnadmin: Malformed representation header

2. Dump the first working section of the repository to a dumpfile, then dump the second section as an incremental dump onto the end of this dumpfile

svnadmin dump myrepo -r0:194 > my.dmp
svnadmin dump myrepo --incremental -r196:HEAD >> my.dmp

3. Try to rebuild this repository by creating a new repository, and using svnadmin load to read in the newly created dumpfile.

svnadmin create newrepo
cat my_filtered.dmp | svnadmin load newrepo

4. It’s likely that paths can not be rebuilt, and you will see errors describing this. Use svndumpfilter to exclude the problematic paths, creating a new dumpfile for each exclusion, and repeating from step 3.

cat my.dmp | svndumpfilter exclude "My/Bad/Path" > my_filtered.dmp
Excluding prefixes:
   '/My/Bad/Path'

Revision 0 committed as 0. (etc)

5. Replace the rebuilt repository’s UUID with that of the old repository, by using svnlook uuid and svnadmin setuuid.

svnadmin setuuid newrepo `svnlook uuid myrepo`

6. Use svn switch to attempt to move a checked out copy to your new repository.

svn --relocate switch http://host/myrepo http://host/newrepo .

7. If all goes to plan, replace your corrupt repository with your rebuild, and svn switch any working copies back to their original path. Then copy back in the data that you lost in the exclusion process, commit and get on with your work.

There you go, loads of messing around to get back to square one. Once that’s done, stop using FSFS as your SVN backend, as it’s much more buggy!

The modern concept of ‘web services’ has always existed in one form or another. Originally the protocols were binary and incredibly complex to integrate, but times have changed a lot. However, there are still large swathes of service providers who provide web-based APIs, but only support primitive versions of these- relying on ‘file drops’ and simple callbacks. Worse still, these providers are particularly resistant to changing their ways. This can be a serious bar of entry for more modern development companies who are used to a SOAP/REST based system, stopping them climb the supplier chain.

This short example shows how you can use PHP to integrate with a supplier who insists on using a file drop mechanism – in this case, to get your date-stampped CSV files, prefixed “file-” wrapped up in ZIP file using the format user-DMY, on an FTP site of their choice!

$host='www.example.com';
$user='myuser';
$pass='mypass';
$date=date('dmy');

// where to store file
$store='/tmp/files.zip';

// get file
$ch = curl_init('ftp://{$user}:{$pass}@{$host}/{$user}-{$date}.zip');
curl_setopt($ch,CURLOPT_RETURNTRANSFER,1);
$file=curl_exec($ch);
file_put_contents($store,$file);

// output ids of all entries
$entries = getEntriesFromCSV(getEntryFile($store) );
foreach($entries as $entry){
    print '{$entry['id']}\n';
}

/*
* @desc   Get CSV number file from local ZIP file
* @author Leo Brown
* @param  $file String Local file to open
*/
function getEntryFile($file)
{
    if ($file &
    &
    is_resource($zip = zip_open($file))) {
        while ($entry=zip_read($zip)) {
            if ('file' == substr(zip_entry_name($entry),0,4)) {
                return zip_entry_read($entry, zip_entry_filesize($entry
                )
                );
            }
        }
    }
}

/*
* @desc   Get provider entries from CSV formatted string
* @author Leo Brown
* @param  $file String Allocated entries as 'entry,t1,t2,t3\n'
*/
function getEntriesFromCSV($file)
{
    $lines=explode('\n',$file);
    $entries=array();
    foreach($lines as $line){
        $fields=explode(',',trim($line));
        if (is_numeric($fields[0])) {
            $entry=array('id'=>$fields[0],
            'data1'=>@$fields[1],
            'data2'=>@$fields[2],
            'data3'=>@$fields[3]
            );
            $entry[$fields[0]]=array_filter($entry);
        }
    }
    return $entries;
}
?>

Today Acumen launched the new Peace Direct website.

Peace Direct’s cause is grassroots peacebuilding initiatives, and is the sister site of Acumen’s earlier project Insight on Conflict.

The site was built on Wordpress and utilises some of the latest features to create a great-looking, dynamic and easily maintainable site. You can access the site at the following link:

http://www.peacedirect.org

Acumen today launched the new website for the Centre for Investigative Journalism, a training and research charity committed to education and reporting based in London.

The site is built in WordPress, the open-source CMS, and is produced from designs provided by Typographics .

Visit the TCIJ site for more info: http://www.tcij.org.

Today Acumen launched a new service allowing Touchnote users to generate cards on the basis of their Facebook photos.

Try out the new app at http://www.touchnote.com/facebook

Article Cuttings:

• Touchnote launches API, integrates with Facebook photos
• Touchnote provides direct access to Facebook albums

Despite great Evangelism about the breakthroughs of Web2.0, the design limitations of HTML still live on – a total lack of CSS3 uptake, and a highly restrictive set of web fonts – barely three serifs and sans serifs to match. While possibilities with content and navigation have been opened up, design limitations of standard web pages are leaving DHTML standing for Dilapidated HTML, and has driven so many style-conscious sites to be poorly implemented in flash – which due to a lack of complex positioning tools can be very ignorant of viewport size. The birth of sIFR was a breakthrough crossover between Flash and HTML, but this still comes with some restrictions and risks that many companies find hard to justify on their sites.

At Acumen, we’ve never wanted to restrain designers too much by forcing them to use web fonts, but then we also take on a responsibility to offer our customers sites that they can fully control. Matting down a title font for each page makes it hard for webmasters without a design background to make the changes they need, and so we’ve used a number of tools along the way to bring that control back to the maintainers.

Key among these is a system to render non-webfonts on the fly, determined by user-generated content. You can see this in use on the Acumen site, for instance the use of Bank Gothic on the title of this page. This is accomplished using a WordPress plugin we’ve written to replace specific page tags and CSS selectors – and it works fairly seemlessly, requiring only Javascript and PHP to run.

Anyway, by request, we’ve decided to release the plugin for general use – enjoy!

Acumen are proud to report the release of the Varsani Richardson website. V+R engaged Acumen to deliver their exact typography requirements to the web, which Acumen accomplished with a range of technologies detailed in previous posts.

You can view the new V+R website at www.varsanirichardsonphoto.com

There are numerous applications for integration of inbound email – the most obvious being ticketing systems, and robot systems (auto/interactive responders).

This is really easily achievable on UNIX systems using the .forward (dot forward) file and your scripting language of choice. If you wanted to pipe all of root’s mail into a script, you would create a .forward file in that user’s home directory, consisting of a single line, starting with a pipe, followed by the full process path.

|/root/process.php

Then you need to provide a destination processor for your email. In this case we have used a UNIX-style file processor at the start by using the hashbang syntax:

#!/usr/local/bin/php
<?php

// read from stdin
$fp = fopen('php://stdin', 'r');
while(!feof($fp)) {
$lines[]=fgets($fp, 4096);
}

[... email processing code...]
?>

You can then use an optional MIME decoder such as Pear mimeDecode to break up the email into the relevant sections – for instance, the from/to fields and plain and/or HTML body text.

While in production applications we all favour use of an API, there are a lot of situations, such as in ‘Mashups’ (I love how that term has been reappropriated from Jungle music) where you need to do some page scraping.

It’s occurred to me how these very easy techniques seem inaccessible to many people, so I thought I’d post a few bits and bobs about some basic scraping methods.

Here’s a bit of code I wrote to use PHP’s DOMDocument class to treat a HTML page as XML and fetch, in this case, the incredibly useful current world population… fantastic!

<?php
 // where to find population data...
 $location['url']='http://www.census.gov/ipc/www/popclockworld.html';
 $location['id']='worldnumber';

 // initialise a new document and prepare the data
 $d=new DOMDocument();
 $file = file_get_contents($location['url']);

 // get and print current world population
 $d->loadHTML($file);
 $e=$d->getElementById($location['id']);
 print $e->nodeValue;
?>

Sample output: 6,738,610,278

Touchnote engaged Acumen to produce the online card creator at the centre of their new product, aimed at users who want to have more control over physical greetings cards sent from the web.

The card creator that Acumen built allows the uploading of images from a user’s computer, subsequent resizing and positioning options, followed by styled captioning and symbols both on the front and inside of the card.

Today the Touchnote product was released, and we wish them all the best for the coming months!

You can access Touchnote at www.touchnote.com.