Sunday, August 30, 2009

F# - Asynchronous Directory.GetFiles and File.Copy

The Problem:
My wife recently decided to take on the task of ordering family photo albums. This primarily involved wading through the deluge of digital photos that have been piling up over the last three years, selecting the photos that were worthy to be placed in the album, uploading those photos to a certain web site, organizing the photos in the albums, and finally making the purchase. The particular imaging software, that was used to import/organize/modify the photos, stores the images in a directory structure that looks something like this ..\<Main Directory>\<Year>\<Month>\<Day>. While this would not normally be an issue, it made the task of uploading the images to the photo album creation site quite arduous.

This of course is a simple problem to solve in most programming languages. The following examples show how it could be solved with F# both synchronously and asynchronously. While this is definitely not production ready, it provides a few samples of how asynchronous workflows can be used in the wild.

The Synchronous Way:
(Note: A sleep statement has been added to emphasize the speed difference between the synchronous/asynchronous approaches.)
open System.IO
let destinationDirectory = @"C:\temp\picDest\"
let sourceDirectoryRoot = @"C:\temp\pic"
let searchPattern = @"*.jpg";
let getFileName sourceFile = FileInfo(sourceFile).Name
let getSourceImages = Directory.GetFiles(sourceDirectoryRoot, searchPattern, 
                          SearchOption.AllDirectories)
let getDestinationFileName sourceFile destinationDirectory = 
    destinationDirectory + getFileName sourceFile
let copyImage sourceFile destinationDirectory = 
    File.Copy(sourceFile, getDestinationFileName sourceFile destinationDirectory, true) 
    |> ignore  

do printfn 
    "Starting the image consolidation process with base directory: %s" sourceDirectoryRoot
for image in getSourceImages do
    System.Threading.Thread.Sleep(5000)
    do printfn "[.NET Thread %d] %s" System.Threading.Thread.CurrentThread.ManagedThreadId image
    copyImage image destinationDirectory
do printfn "The images have been consolidated into directory %s." destinationDirectory
do printfn "Press [Enter] close this command prompt." 
System.Console.ReadKey() |> ignore
The Asynchronous Way:
(Note: A sleep statement has been added to emphasize the speed difference between the synchronous/asynchronous approaches.)
open System
open System.IO

type Directory with
    static member AsyncGetFiles(path:string, searchPattern:string, searchOption:SearchOption) = 
        let fn = new Func <string * string * SearchOption, string[]>(Directory.GetFiles)    
        Async.BuildPrimitive((path, searchPattern, searchOption), fn.BeginInvoke, fn.EndInvoke)
type File with
    static member AsyncCopy(sourceFile:string, destinationFile:string, overwrite:bool) = 
        let fn = new Func<string * string * bool, unit>(File.Copy)
        Async.BuildPrimitive((sourceFile, destinationFile, overwrite), fn.BeginInvoke, fn.EndInvoke)   

let destinationDirectory = @"C:\temp\picDest\"
let sourceDirectoryRoot = @"C:\temp\pic"
let searchPattern = @"*.jpg";
let getFileName sourceFile = FileInfo(sourceFile).Name
let getSourceImages imageDirectory searchPattern searchOption = 
    async { return! Directory.AsyncGetFiles(imageDirectory, searchPattern, 
                searchOption) }
let getDestinationFileName sourceFile destinationDirectory = 
    destinationDirectory + getFileName sourceFile
let copyImage sourceFile destinationDirectory overwrite = 
    async {
        System.Threading.Thread.Sleep(5000)
        do printfn 
            "[.NET Thread %d] %s" Threading.Thread.CurrentThread.ManagedThreadId sourceFile
        return! File.AsyncCopy(sourceFile, 
            getDestinationFileName sourceFile destinationDirectory, overwrite) }

do printfn 
    "Starting the image consolidation process with base directory: %s" 
    sourceDirectoryRoot

let sourceImages = 
    getSourceImages sourceDirectoryRoot searchPattern SearchOption.AllDirectories
    |> Async.RunSynchronously

for image in sourceImages do
    copyImage image destinationDirectory true
    |> Async.Start
       
do printfn 
    "The images will be consolidated into the following directory: %s" destinationDirectory 
System.Console.ReadKey() |> ignore

18 comments:

  1. A few questions:
    - What's the advantage of having getSourceFiles be async?
    - What is the exact interaction between Async.Start and the threadpool -- I assume Async.Start will block if there are no threadpool threads available? Even so, this is probably too many threads, no?

    You mention hilighting the speed differrences between the approaches, but make no mention of them - what were your findings? (Without the sleep() - that's cheating :))

    I'd expect the async copy to not be much, or any faster, as the copy operation is I/O bound, and a busy i/o channel and added disk head movement could actually slow things down.

    thanks

    ReplyDelete
  2. China and Russia put the blame on some screwed up experiments of US for the earthquake that happened in Haiti.
    Chinese and Russian Military scientists, these reports say, are concurring with Canadian researcher, and former Asia-Pacific Bureau Chief of Forbes Magazine, Benjamin Fulford, who in a very disturbing video released from his Japanese offices to the American public, details how the United States attacked China by the firing of a 90 Million Volt Shockwave from the Americans High Frequency Active Auroral Research Program (HAARP) facilities in Alaska
    If we can recollect a previous news when US blamed Russia for the earthquake in Georgio. What do you guys think? Is it really possible to create an earthquake by humans?
    I came across this [url=http://universalages.com/hot-news/what-happened-in-haiti-is-it-related-to-haarp/]article about Haiti Earthquake[/url] in some blog it seems very interesting, but conspiracy theories have always been there.

    ReplyDelete
  3. China and Russia put the blame on some screwed up experiments of US for the earthquake that happened in Haiti.
    Chinese and Russian Military scientists, these reports say, are concurring with Canadian researcher, and former Asia-Pacific Bureau Chief of Forbes Magazine, Benjamin Fulford, who in a very disturbing video released from his Japanese offices to the American public, details how the United States attacked China by the firing of a 90 Million Volt Shockwave from the Americans High Frequency Active Auroral Research Program (HAARP) facilities in Alaska
    If we can recollect a previous news when US blamed Russia for the earthquake in Georgio. What do you guys think? Is it really possible to create an earthquake by humans?
    I came across this [url=http://universalages.com/hot-news/what-happened-in-haiti-is-it-related-to-haarp/]article about Haiti Earthquake[/url] in some blog it seems very interesting, but conspiracy theories have always been there.

    ReplyDelete
  4. New here,

    I'm just browsing sites for the children of Haiti.

    I'm doing my part for a non-profit organization that gives time to
    creating an oppurunity for the kids in haiti. If anybody wants to give money then this is the place:

    [url=http://universallearningcentre.org]Donate to Haiti[/url] or Help Haiti

    They provide children in Haiti a positive learning environment.

    Please check it out, they're real.

    Please give

    ReplyDelete
  5. Genial post and this enter helped me alot in my college assignement. Thanks you for your information.

    ReplyDelete
  6. [COLOR="Red"][B]Click on the pictures to view in full size[/B][/COLOR]

    [URL=http://www.wallpaperhungama.in/details.php?image_id=12779][IMG]http://www.wallpaperhungama.in/data/media/32/Asin-80.jpg[/IMG][/URL]

    [URL=http://www.wallpaperhungama.in/details.php?image_id=7237][IMG]http://www.wallpaperhungama.in/data/media/32/Asin-71.jpg[/IMG][/URL]


    [url=http://www.wallpaperhungama.in/cat-Asin-32.htm][b]Asin Hot Wallpapers[/b][/url]

    Photo gallery at t WallpaperHungama.in is dedicated to Asin Pictures. Click on the thumbnails into enlarged Asin pictures, intimate photographs and exclusive photos. Also after to other Pictures Gallery for Turbulent quality and High Acutance twin scans, talking picture captures, movie promos, wallpapers, hollywood & bollywood pictures, photos of actresses and celebrities

    ReplyDelete
  7. like gambling? love las vegas? invasion the all unpractised in [url=http://www.casinolasvegass.com]casino[/url] las vegas at www.casinolasvegass.com with all across and beyond 75 proper to distribution unstinting [url=http://www.casinolasvegass.com]online casino[/url] games like slots, roulette, baccarat, craps and more and sieve a crushing in licit coins with our $400 latest crazed bonus.
    we from teeth of all things safer games then the superannuated online [url=http://www.place-a-bet.net/]casino[/url] www.place-a-bet.net! few more free casino games and free [url=http://www.2010-world-cup.info]casino bonus[/url] you can find at the 2 new [url=http://www.buy-cheap-computers.info]casino[/url] guides : www.2010-world-cup.info and www.buy-cheap-computers.info .

    ReplyDelete
  8. Hi readers
    Do not miss your chance to get a free ipad. Visit http://bit.ly/cmmVr7

    ReplyDelete
  9. Online guitar lessons are a great way to learn guitar. Cheaper and more convenient than private studio lessons, the technology is now at a point where the difference in quality is nonexistent and actually has many advantages.
    [url=http://www.goarticles.com/cgi-bin/showa.cgi?C=2514060]Online Guitar Lessons[/url]

    ReplyDelete
  10. The French gourmet cheese Bleu d'Auvergne has a wonderful aroma, a rich taste; the saltiness increases with the incidence of veining. The overall flavor is piquant but not overly sharp. Bleu d'Auvergne started life as an imitation of Roquefort, using cow's milk in place of sheep's milk. Legend has it that a peasant, around 1845, decided to inject his cheese with a blue mold that he found growing on his left-over bread (the motto being, waste not, want not). And thus, the gourmet cheese Bleu d'Auvergne was born. This French gourmet blue cheese comes from the region of Auvergne and the cheese is made from milk of Salers and Aubrac cows. The rind is very thin and so the cheese is usually wrapped in foil. The cheese is rich and creamy with a pale yellow color and scattered holes and well-defined greenish-blue veining. We cut and wrap this cheese in wedge of 8 ounces and 1 pound.

    buy fresh blue cheese

    [url=http://riderx.info/members/buy_5F00_fresh_5F00_blue_5F00_cheese.aspx]buy fresh blue cheese[/url]

    http://riderx.info/members/buy_5F00_fresh_5F00_blue_5F00_cheese.aspx

    ReplyDelete
  11. Hello,
    I have developed a new clean web 2.0 wordpress theme.

    Has 2 colours silver and blue, has custom header(colour or image).
    I am curently working on it, so if you have suggestions let me know.

    You can view live demo and download from here www.getbelle.com
    If you found bug reports or you have suggestions pm me.
    Wish you a happing using.

    many thanks to [url=http://www.usainstantpayday.com/]USAInstantPayDay.com[/url] for paying the hosting and developement of the theme

    Fumboppophy

    ReplyDelete
  12. Hello friend amazing and very interesting blog about F# - Asynchronous Directory.GetFiles and File.Copy

    ReplyDelete
  13. Generic Viagra contains Sildenafil and it is a prescription drug used to treat erectile dysfunction. Erectile dysfunction is the inability of the penis to become rigid, or stay firm long enough to complete the sexual act.Generic Viagra is used to increase blood flow to certain areas of the body, as a muscle relaxant and as a male impotence or erectile dysfunction treatment. Generic Propecia is a one-a-day, FDA-approved, treatment for male pattern hair loss at the crown or in the middle of the scalp (androgenetic alopecia). Generic Viagra is the world renowned medicine for erectile dysfunction and most men have forgotten they ever had a problem with erections after its use. Take it on empty stomach and it will work in 30 minutes. Erections can be achieved on stimulation.Generic Cialis is consumed orally and has quickly become one of the most prescribed medications. Generic Levitra has been clinically proven to improve erectile function, even in men who had other health factors, like high cholesterol, high blood pressure, or diabetes

    ReplyDelete
  14. Hello I want to congratulate to them by its site of the Web of the excellent looks like entertained and very good very to me it elaborated.

    ReplyDelete
  15. Hello I want to congratulate to them by its site of the Web of the excellent looks like entertained and very good very to me it elaborated.

    ReplyDelete
  16. Hi readers
    Do not miss your chance to get a free ipad. Visit http://bit.ly/cmmVr7

    ReplyDelete
  17. Comprar mandarinas por internet en www.mandarinasfrescas.com. Venta de mandarinas online directamente del agricultor al consumidor. La mejor mandarina al mejor precio.
    Venta online directa de mandarinas dulces valencianas, entregamos del campo a tu domicilio. Para consumir como fruta o zumo, ¡descúbrenos!
    Ahora puedes comprar las autenticas mandarinas de Valencia por Internet. Mandarinas frescas de la mejor calidad, directamente del arbol a tu casa en 24 horas.
    Venta mandarinas a domicilio internet, comprar online, Castellón. Clementinas por internet, mandarinas internet.
    Mandarinas valencianas recien recolectadas. Envíos a domicilio.
    Venta por internet de mandarinas naturales, directas del agricultor sin intermediarios.
    Naranjas online. Venta de mandarinas. Mandarinas de calidad Extra, fruta gourmet de excelente sabor al mejor precio.
    Venta de mandarinas a domicilio por Internet. Compra mandarinas valencianas a domicilio OnLine. Garantia y Calidad. Origen Valencia, Mandarinas y Frutas.

    ReplyDelete