A Few New NoSQL Helper Libraries

I've been working on a few new libraries lately that focus on providing functional wrappers around various NoSQL options.

FSharpCouch:

FSharpCouch is something that I wrote about a little over two years ago. At the time, it was more of an educational exercise, but I have since revised it and found aspects of it to be useful. This is now available on NuGet as package ID FSharpCouch.

Here's an example:


You can find additional examples and the full source at https://github.com/dmohl/FSharpCouch.

MongoFs:

MongoFs currently has a primary focus on function composition as it relates to setting up the MongoDB client aspects of server, database, and collection. It simply wraps functions around the necessary methods of the C# Mongo Driver and auto opens the encompassing module.

Here's an example:


You can get it from NuGet as package ID MongoFs and find more examples and source at https://github.com/dmohl/MongoFs.

FSharpCouch: A Simple CouchDB .NET API in F#

In one of my last posts I showed a simple web application (created with the help of WebSharper Platform 2010) that captured registration information from a user and saved it to a CouchDB database.  I've been using CouchDB for a while now and I'm still amazed at how quickly you can get things up and running when you don't have to worry about the object-relational impedance mismatch.

To interact with CouchDB, I primarily use Relax (a full featured ".NET API abstraction of CouchDB's (excellent) RESTful API"); however, I thought it might be fun and educational to create a simple CouchDB .NET API abstraction in F#.  Note: The code provided  here is loosely based on SharpCouch.

Just Show Me the Code:

Without any further ado, here's the code.

module FSharpCouch
    open System
    open System.Net
    open System.Text
    open System.IO
    open Newtonsoft.Json
    open Newtonsoft.Json.Linq

    let WriteRequest url methodName contentType content =
        let request = WebRequest.Create(string url)
        request.Method <- methodName
        request.ContentType <- contentType 
        let bytes = UTF8Encoding.UTF8.GetBytes(string content)
        use requestStream = request.GetRequestStream()
        requestStream.Write(bytes, 0, bytes.Length) 
        request
    let AsyncReadResponse (request:WebRequest) =
        async { use! response = request.AsyncGetResponse()
                use stream = response.GetResponseStream()
                use reader = new StreamReader(stream)
                let contents = reader.ReadToEnd()
                return contents }
    let ProcessRequest url methodName contentType content =
        match methodName with
        | "POST" -> 
            WriteRequest url methodName contentType content
        | _ -> 
            let req = WebRequest.Create(string url)
            req.Method <- methodName
            req
        |> AsyncReadResponse 
        |> Async.RunSynchronously
    let ToJson content =
        JsonConvert.SerializeObject content
    let FromJson<'a> json =
        JsonConvert.DeserializeObject<'a> json
    let BuildUrl (server:string) (database:string) =
        server + "/" + database.ToLower()
    let CreateDatabase server database =
        ProcessRequest (BuildUrl server database) "PUT" "" ""
    let DeleteDatabase server database =
        ProcessRequest (BuildUrl server database) "DELETE" "" ""
    let CreateDocument server database content = 
        content |> ToJson
        |> ProcessRequest (BuildUrl server database) "POST" "application/json"
    let GetDocument<'a> server database documentId =
        ProcessRequest ((BuildUrl server database) + "/" + documentId) "GET" "" ""
        |> FromJson<'a>
    let GetAllDocuments<'a> server database =
        let jsonObject = ProcessRequest ((BuildUrl server database) + "/_all_docs") "GET" "" ""
                         |> JObject.Parse
        Async.Parallel [for row in jsonObject.["rows"] -> 
                            async {return FromJson<'a>(row.ToString())}]
        |> Async.RunSynchronously
    let DeleteDocument server database documentId revision =         
        ProcessRequest ((BuildUrl server database) + "/" + documentId + "?rev=" + revision) "DELETE" "" ""

Conclusion:

F# + CouchDB = totally awesome! You can find the full solution with integration tests at http://github.com/dmohl/FSharpCouch.

Getting Started with WebSharper Platform 2010

A little while ago a cool new product was brought to my attention that allows client-based web development in F#.  The name of this product is WebSharper Platform 2010.  In this post, I will show a simple example for getting started with WebSharper Platform 2010.

Overview of this Example: 
The following code is a slimmed down and slightly modified version of the example provided in the default WebSharper-1.0 template.  Additionally, this example saves the information captured on the form to a CouchDB database.  Note: To run this example, you will need to download and install the WebSharper Platform 2010 Standard - Version 1.0.28 - Public Release (Instructions can be found at http://www.intellifactory.com/products/wsp/GettingStarted.aspx).  Additionally, in order for the registration information to be saved to the CouchDB database, CouchDB must be running on the default port (5984) of the local machine and a database named registration must exist.

Setting up the Form:
Our basic form is setup with the help of the Formlets library ("a library for type-safe web form combinators" - http://www.intellifactory.com/docs/websharper.pdf). The JavaScript attribute informs the WebSharper compiler that the function should be translated from F# to JavaScript.

    [<JavaScript>]
    let RegistrationForm : Formlet<RegistrationInformation> =
        Formlet.Yield (fun firstName lastName email -> 
                           {FirstName = firstName; 
                            LastName = lastName; Email = email})
        <*> input "First Name" "Please enter your first name"
        <*> input "Last Name" "Please enter your last name"
        <*> inputEmail "Email" "Please enter a valid email address"

Creating the Form Validation:
You probably noticed the input and inputEmail function calls in the form setup. These functions create the form elements with appropriate validation.  The code is as follows:

    [<JavaScript>]
    let input (label: string) (error: string) = 
        Controls.Input ""
        |> Validator.IsNotEmpty error
        |> Enhance.WithValidationIcon
        |> Enhance.WithTextLabel label

    [<JavaScript>]
    let inputEmail (label: string) (error: string) = 
        Controls.Input ""
        |> Validator.IsEmail error
        |> Enhance.WithValidationIcon
        |> Enhance.WithTextLabel label

Adding the Flow:
In this simple example, the user completes the three fields and clicks submit. The information is then saved to couch and the user is redirected to a summary page. That sequence is setup with the following code:

    [<JavaScript>]
    let RegistrationSequence =
        let registrationForm =
            RegistrationForm
            |> Enhance.WithSubmitAndResetButtons
            |> Enhance.WithCustomFormContainer {
                Enhance.FormContainerConfiguration.Default with
                    Header = 
                        "Enter the following information to register:" 
                        |> Enhance.FormPart.Text 
                        |> Some
                }
        let completeRegistration registrationInformation () =
            SaveRegistrationToCouch registrationInformation
            FieldSet [
                Legend [Text "Registration summary"]
                P ["Hi " + registrationInformation.FirstName + " " + 
                    registrationInformation.LastName + "!" |> Text]
                P ["You are now registered." |> Text]]
        let flow =
            Flowlet.Do {
                let! initialForm = registrationForm
                return! Formlet.OfElement (completeRegistration initialForm)
            }
        Div [H1 [Text "Register today!"]] -< [flow.BuildForm()]

Saving the Information to CouchDB: 
The final piece of code that I will show from this example is the server side method that is called from the client to save the information to CouchDB. In the RegistrationSequence function, we saw a call to a function named SaveRegistrationToCouch. This function looks like this:

    [<Rpc>]
    let SaveRegistrationToCouch (registrationInformation:RegistrationInformation) =
        JsonConvert.SerializeObject registrationInformation
        |> FSharpCouch.CreateDocument couchDBUrl 
        |> ignore

As you can see, all it takes for us to expose a server-side method is to add the Rpc attribute to that function.

Viewing the Result:
The result is a nice little registration sequence that contains client-side validation.  An example of the initial form with all completed information is below:

Conclusion:
This example shows how easy it is to throw together a simple registration form with WebSharper Platform 2010. We've only touched the surface on the capabilities of this platform.  You can find out more by visiting the Intellifactory web site.  You can find the full solution at http://github.com/dmohl/WebSharper-Example.

Integrating F# and Erlang Using OTP.NET

Recently, I ran across a post that talked about a .NET library named OTP.NET.  OTP.NET is a port of a Java library named JInterface that allows a .NET application to act as an Erlang node.  OTP.NET hasn't been updated in a while and the API is a bit clunky, but the provided functionality is very cool.  Note: A forked version of OTP.NET that has been migrated to VS2010 can be found at http://github.com/dmohl/OTP.NET-2010.

I decided to give OTP.NET a test drive by adding a simple command-line interface to an Erlang project that I have been playing with called Querly.

Querly is a simple query engine that allows T-SQL style queries against a CouchDB database.  You can read more about Querly in the following posts:

- Querly - A Query Engine for CouchDB
- Setting Up Querly
- Getting Started with Querly: A Simple CouchDB Query Engine

Getting Setup


To make this work, we first need to do the following:

1. Setup Querly by following the instructions provided in the blog post entitled Setting Up Querly.
2. If you want to load up a database for testing, follow the instructions provided in the blog post entitled Getting Started with Querly: A Simple CouchDB Query Engine.
3. Make sure that RabbitMQ and CouchDB are running.
4. Launch Querly by navigating to the Querly directory and running a command such as:
"C:\Program Files\erl5.7.5\bin\werl.exe" -sname querly -setcookie supersecretcookie -pa ./ebin -pa ./src
5. Finally, in the Erlang interactive shell, run the following command:
querly:start().

If everything worked as expected, you should see something like this:

Building the F# Application

Now that we have the Erlang application up and running, it's time to put together the F# code that will allow us to join the Erlang node cluster and interact with Querly.  Note: You'll need to replace the value assigned to peerNode with the name of your querly node. The full solution can be found at http://github.com/dmohl/Ferly.

module Ferly

open System
open Otp

Console.Write ":>"  
let mutable input = Console.ReadLine()  

let erlangCookie = "supersecretcookie"
let peerNode = "querly@dmohl-PC" // Replace with querly node name

let connection =
    let cNode = new OtpSelf("querlyclientnode", erlangCookie)
    let sNode = new OtpPeer(peerNode)
    cNode.connect(sNode)

let RemoteCall (sql:string) = 
    let arguments = [|new Erlang.String(sql)|] : Erlang.Object[]
    do connection.sendRPC("querly_client", "sql_query", arguments) 
    connection.receiveRPC().ToString()

while input <> "quit" do 
    Console.Write(RemoteCall input)
    Console.Write "\n\n:>"   
    input <- Console.ReadLine()

We should now be able to run the F# application and write queries against a couch database.  An example of the running application after a few processed queries is shown below:

Wrapping Up


We have only touched the surface of the functionality provided by OTP.NET.  Hopefully this post has shown how OTP.NET can be used to quickly integrate F# with Erlang.  Unfortunately, there is not a lot of documentation out there for OTP.NET, so if you want to learn more, your best bet is to pull down the code and start thumbing through it.  Happy polyglot programming!

Getting Started with Querly: A Simple CouchDB Query Engine

About a month ago, I introduced a simple project that allows limited T-SQL style queries against a CouchDB database.  The codename for that project is Querly.  Today, I'm going to talk a little about what you need to get started with Querly.

The other posts can be found in the following locations:

- Querly - A Query Engine for CouchDB  
- Setting Up Querly

Loading Up A Test Database

To get started with Querly, make sure that you have followed the instructions outlined in Setting Up Querly.  If all the tests run, you are ready to get started with Querly.  Press [Enter] to get back to the Erlang prompt.  In order to load up a database with several documents that we can play with, type  querly_load_tests:load_10000_people(). and press [Enter] (Note: This will take a little while to finish).

Writing Our First Query

Once the "person" database is loaded, we can start Querly and begin writing simple queries.

To start querly, type querly:start().

Here are a few samples of queries that can now be run (Note: The first query may take a few seconds as the documents are being loaded into an ETS table):

querly_client:sql_query("select * from person where Idno = 1").

querly_client:sql_query("select * from person where FirstName = TestFirst2").

querly_client:sql_query("select * from person where Dob= 08/28/1977 and FirstName = TestFirst3").

Querying Other Databases

In order to write T-SQL style queries against a database, a record must be defined in the src/record_definitions.hrl file.  An example for the person table is as follows:

-record(person, {'Idno', 'FirstName', 'LastName', 'Dob', 'Ssn'}). 

-define(personFields, record_info(fields, person)).

This definition makes it possible to use specific field names in the TSQL style queries.

Limitations

There are a number of known limitations in this simple query engine. 

The most notable limitations are as follows:

- The T-SQL style query functionality is limited to simple "where" clauses.  

- A record must be predefined in order to query a database.

Conclusion

Querly allows simple T-SQL style queries against a CouchDB database.  While Querly has a number of known limitations, it hopefully serves as a simple PoC and/or starting point for a CouchDB query engine.

Interacting with RabbitMQ via F# and Symbiote

Last week, I showed how easy it is to talk to CouchDB with F# and Symbiote.  In this post, I'll show how you can start interacting with RabbitMQ by adding an additional dozen or so lines of code.

RabbitMQ:

RabbitMQ is a complete and highly reliable enterprise messaging system based on the emerging AMQP standard. It is licensed under the open source Mozilla Public License and has a platform-neutral distribution, plus platform-specific packages and bundles for easy installation. (http://www.rabbitmq.com/)

Symbiote:

Symbiote is a set of libraries that reduce the radius of comprehension by providing simplified APIs "...to some of the better open source libraries available" as well as "...a way to centralize the configuration [and] dependency injection..." of said libraries (Robson, A., Symbiote).   

For those of you who haven't played with RabbitMQ and/or Symbiote, I strongly recommend checking them out!

Code:

Here's the code with the CouchDB interaction functionality as well as the RabbitMQ publish/subscribe functionality.

A couple of things to note about this sample:
1. CouchDB and RabbitMQ must be installed and running on the local machine.
2. Since Symbiote is under heavy development, the CouchDB related code has changed slightly from the example provided last week.

module Progam

open System
open Symbiote.Core
open Symbiote.Relax
open Symbiote.Daemon
open Symbiote.Jackalope
open Symbiote.Jackalope.Impl

type PersonCouchDocument =
    val id : string
    val mutable name : string
    val mutable address : string
    inherit DefaultCouchDocument 
        new (id, name, address) = 
            {id = id; name = name; address = address} then base.DocumentId <- id
        member x.Name
            with get () = x.name
        member x.Address
            with get () = x.address

type HandleMessage() =
    interface IMessageHandler<PersonCouchDocument> with
        member x.Process (message:PersonCouchDocument, messageDelivery:IMessageDelivery) =
            Console.WriteLine("Processing message for person {0}.", message.name)
            messageDelivery.Acknowledge()

type DemoService = 
    val couchServer : ICouchServer
    val bus : IBus
    new(couchServer, bus) as this = 
        {couchServer = couchServer; bus = bus} then this.Initialize()
    member public x.Initialize () =
        x.bus.AddEndPoint(fun x -> 
            x.Exchange("SymbioteDemo", ExchangeType.fanout).QueueName("SymbioteDemo")
                .Durable().PersistentDelivery() |> ignore)
    interface IDaemon with
        member x.Start () =
            do Console.WriteLine("The service has started")
            x.bus.Subscribe("SymbioteDemo", null)
            let document = new PersonCouchDocument("123456", "John Doe", "123 Main")
            x.couchServer.Repository.Save(document)
            x.bus.Send("SymbioteDemo", document)
            let documentRetrieved = 
                x.couchServer.Repository.Get<PersonCouchDocument>(document.DocumentId);
            do Console.WriteLine(
                "The document with name {0} and address {1} was retrieved successfully", 
                documentRetrieved.Name, documentRetrieved.Address)
            do x.couchServer.DeleteDatabase<PersonCouchDocument>()
        member x.Stop () =
            do Console.WriteLine("The service has stopped")

do Assimilate
    .Core()
    .Daemon(fun x -> (x.Name("FSharpDemoService")
                                    .DisplayName("An FSharp Demo Service")
                                    .Description("An FSharp Demo Service")
                                    .Arguments([||]) |> ignore))
    .Relax(fun x -> x.Server("localhost") |> ignore) 
    .Jackalope(fun x -> x.AddServer(fun s -> s.Address("localhost").AMQP08() |> ignore) |> ignore)
    .RunDaemon() 

Conclusion:

As mentioned previously, with the help of Symbiote it only takes a dozen or so additional lines of code to add support for RabbitMQ to our previous example.  The complete source for this solution can be found at http://github.com/dmohl/Symbiote-Jackalope-Example.

Talking to CouchDB via F# and Symbiote

One of the cooler technologies that I have playing with lately is CouchDB.

"Apache CouchDB is a document-oriented database that can be queried and indexed in a MapReduce fashion using JavaScript. CouchDB also offers incremental replication with bi-directional conflict detection and resolution" (http://couchdb.apache.org/).  

Alex Robson recently announced a set of libraries that, among other things, makes talking with CouchDB from a .NET language very easy. Here's an F# example (the complete source can be found at http://github.com/dmohl/Symbiote--Relax--Example):

module Progam

open System
open Symbiote.Core
open Symbiote.Relax
open Symbiote.Daemon

type PersonCouchDocument =
    val id : string
    val mutable name : string
    val mutable address : string
    inherit DefaultCouchDocument 
        new (id, name, address) = 
            {id = id; name = name; address = address} then base.DocumentId <- id
        member x.Name
            with get () = x.name
        member x.Address
            with get () = x.address

type DemoService = 
    val couchServer : ICouchServer
    new(couchServer) = {couchServer = couchServer}
    interface IDaemon with
        member x.Start () =
            do Console.WriteLine("The service has started")
            let document = new PersonCouchDocument("123456", "John Doe", "123 Main")
            x.couchServer.Repository.Save(document)
            let documentRetrieved = 
                x.couchServer.Repository.Get<PersonCouchDocument>(document.DocumentId);
            do Console.WriteLine(
                "The document with name {0} and address {1} was retrieved successfully", 
                documentRetrieved.Name, documentRetrieved.Address)
            do x.couchServer.DeleteDatabase<PersonCouchDocument>()
        member x.Stop () =
            do Console.WriteLine("The service has stopped")

do Assimilate
    .Core()
    .Daemon(fun x -> (x.Name("FSharpDemoService")
                                    .DisplayName("An FSharp Demo Service")
                                    .Description("An FSharp Demo Service")
                                    .Arguments([||]) |> ignore))
    .Relax(fun x -> x.Server("localhost") |> ignore) 
    .RunDaemon() 

As you can see, interacting with CouchDB via F# and Symbiote is a snap.  Symbiote has a large number of additional features and advantages. To learn more about it, go to http://sharplearningcurve.com/wiki/Symbiote-Main.ashx.

Setting Up Querly

In my last post, I introduced Querly (a simple CouchDB query engine).  In this post, I provide the steps for setting up Querly.

1. Start by installing CouchDB.  A windows installer is available on the couchdb wiki.
2. Next, install Erlang.  I'm using version R13B04.
3. Now, install RabbitMQ, available at http://www.rabbitmq.com/download.html.  Note: Querly has built in support for subscription to a RabbitMQ queue.  This allows updates to be received without having to reload an entire database from CouchDB.  This is not required for utilization of the basic functionality of Querly; however, it is required for auto update support and for all of the tests to run successfully.
4. Launch CouchDB and RabbitMQ.
5. Retrieve Querly from http://github.com/dmohl/Querly.
6. Edit make.bat in the Querly directory and change the Erlang directories appropriately.
7. Launch Querly by executing the batch file in the main directory.  This will run all of the tests.

In a future post, I'll talk about how to get started with Querly.

 

Querly - A Query Engine for CouchDB

I've been playing with Erlang and CouchDB for a little while now and have been completely amazed by both.  One of the things that would be very helpful for my team to be able to use CouchDB in more production scenarios would be to have enhanced query functionality.  Querly is the first attempt at this enhancement.  Querly is a solution written in Erlang that allows ETS and/or limited SQL style queries against CouchDB.  While there are a number of limitations with this first attempt, it provides a starting point.  The code is available at http://github.com/dmohl/Querly.

I plan to provide additional information including setup instructions, features, limitations, etc. over my next few blog posts.