Sunday, August 30, 2009

F# - Asynchronous Directory.GetFiles and File.Copy

The Problem:
My wife recently decided to take on the task of ordering family photo albums. This primarily involved wading through the deluge of digital photos that have been piling up over the last three years, selecting the photos that were worthy to be placed in the album, uploading those photos to a certain web site, organizing the photos in the albums, and finally making the purchase. The particular imaging software, that was used to import/organize/modify the photos, stores the images in a directory structure that looks something like this ..\<Main Directory>\<Year>\<Month>\<Day>. While this would not normally be an issue, it made the task of uploading the images to the photo album creation site quite arduous.

This of course is a simple problem to solve in most programming languages. The following examples show how it could be solved with F# both synchronously and asynchronously. While this is definitely not production ready, it provides a few samples of how asynchronous workflows can be used in the wild.

The Synchronous Way:
(Note: A sleep statement has been added to emphasize the speed difference between the synchronous/asynchronous approaches.)
open System.IO
let destinationDirectory = @"C:\temp\picDest\"
let sourceDirectoryRoot = @"C:\temp\pic"
let searchPattern = @"*.jpg";
let getFileName sourceFile = FileInfo(sourceFile).Name
let getSourceImages = Directory.GetFiles(sourceDirectoryRoot, searchPattern, 
                          SearchOption.AllDirectories)
let getDestinationFileName sourceFile destinationDirectory = 
    destinationDirectory + getFileName sourceFile
let copyImage sourceFile destinationDirectory = 
    File.Copy(sourceFile, getDestinationFileName sourceFile destinationDirectory, true) 
    |> ignore  

do printfn 
    "Starting the image consolidation process with base directory: %s" sourceDirectoryRoot
for image in getSourceImages do
    System.Threading.Thread.Sleep(5000)
    do printfn "[.NET Thread %d] %s" System.Threading.Thread.CurrentThread.ManagedThreadId image
    copyImage image destinationDirectory
do printfn "The images have been consolidated into directory %s." destinationDirectory
do printfn "Press [Enter] close this command prompt." 
System.Console.ReadKey() |> ignore
The Asynchronous Way:
(Note: A sleep statement has been added to emphasize the speed difference between the synchronous/asynchronous approaches.)
open System
open System.IO

type Directory with
    static member AsyncGetFiles(path:string, searchPattern:string, searchOption:SearchOption) = 
        let fn = new Func <string * string * SearchOption, string[]>(Directory.GetFiles)    
        Async.BuildPrimitive((path, searchPattern, searchOption), fn.BeginInvoke, fn.EndInvoke)
type File with
    static member AsyncCopy(sourceFile:string, destinationFile:string, overwrite:bool) = 
        let fn = new Func<string * string * bool, unit>(File.Copy)
        Async.BuildPrimitive((sourceFile, destinationFile, overwrite), fn.BeginInvoke, fn.EndInvoke)   

let destinationDirectory = @"C:\temp\picDest\"
let sourceDirectoryRoot = @"C:\temp\pic"
let searchPattern = @"*.jpg";
let getFileName sourceFile = FileInfo(sourceFile).Name
let getSourceImages imageDirectory searchPattern searchOption = 
    async { return! Directory.AsyncGetFiles(imageDirectory, searchPattern, 
                searchOption) }
let getDestinationFileName sourceFile destinationDirectory = 
    destinationDirectory + getFileName sourceFile
let copyImage sourceFile destinationDirectory overwrite = 
    async {
        System.Threading.Thread.Sleep(5000)
        do printfn 
            "[.NET Thread %d] %s" Threading.Thread.CurrentThread.ManagedThreadId sourceFile
        return! File.AsyncCopy(sourceFile, 
            getDestinationFileName sourceFile destinationDirectory, overwrite) }

do printfn 
    "Starting the image consolidation process with base directory: %s" 
    sourceDirectoryRoot

let sourceImages = 
    getSourceImages sourceDirectoryRoot searchPattern SearchOption.AllDirectories
    |> Async.RunSynchronously

for image in sourceImages do
    copyImage image destinationDirectory true
    |> Async.Start
       
do printfn 
    "The images will be consolidated into the following directory: %s" destinationDirectory 
System.Console.ReadKey() |> ignore

8 comments:

  1. A few questions:
    - What's the advantage of having getSourceFiles be async?
    - What is the exact interaction between Async.Start and the threadpool -- I assume Async.Start will block if there are no threadpool threads available? Even so, this is probably too many threads, no?

    You mention hilighting the speed differrences between the approaches, but make no mention of them - what were your findings? (Without the sleep() - that's cheating :))

    I'd expect the async copy to not be much, or any faster, as the copy operation is I/O bound, and a busy i/o channel and added disk head movement could actually slow things down.

    thanks

    ReplyDelete
  2. New here,

    I'm just browsing sites for the children of Haiti.

    I'm doing my part for a non-profit organization that gives time to
    creating an oppurunity for the kids in haiti. If anybody wants to give money then this is the place:

    [url=http://universallearningcentre.org]Donate to Haiti[/url] or Help Haiti

    They provide children in Haiti a positive learning environment.

    Please check it out, they're real.

    Please give

    ReplyDelete
  3. Genial post and this enter helped me alot in my college assignement. Thanks you for your information.

    ReplyDelete
  4. [COLOR="Red"][B]Click on the pictures to view in full size[/B][/COLOR]

    [URL=http://www.wallpaperhungama.in/details.php?image_id=12779][IMG]http://www.wallpaperhungama.in/data/media/32/Asin-80.jpg[/IMG][/URL]

    [URL=http://www.wallpaperhungama.in/details.php?image_id=7237][IMG]http://www.wallpaperhungama.in/data/media/32/Asin-71.jpg[/IMG][/URL]


    [url=http://www.wallpaperhungama.in/cat-Asin-32.htm][b]Asin Hot Wallpapers[/b][/url]

    Photo gallery at t WallpaperHungama.in is dedicated to Asin Pictures. Click on the thumbnails into enlarged Asin pictures, intimate photographs and exclusive photos. Also after to other Pictures Gallery for Turbulent quality and High Acutance twin scans, talking picture captures, movie promos, wallpapers, hollywood & bollywood pictures, photos of actresses and celebrities

    ReplyDelete
  5. Hello,
    I have developed a new clean web 2.0 wordpress theme.

    Has 2 colours silver and blue, has custom header(colour or image).
    I am curently working on it, so if you have suggestions let me know.

    You can view live demo and download from here www.getbelle.com
    If you found bug reports or you have suggestions pm me.
    Wish you a happing using.

    many thanks to [url=http://www.usainstantpayday.com/]USAInstantPayDay.com[/url] for paying the hosting and developement of the theme

    Fumboppophy

    ReplyDelete
  6. Hello friend amazing and very interesting blog about F# - Asynchronous Directory.GetFiles and File.Copy

    ReplyDelete
  7. Hello I want to congratulate to them by its site of the Web of the excellent looks like entertained and very good very to me it elaborated.

    ReplyDelete
  8. Hello I want to congratulate to them by its site of the Web of the excellent looks like entertained and very good very to me it elaborated.

    ReplyDelete