X
X

Login

Login if you already have an account

LOGIN

Register

Create a new account. join us!

REGISTER

Support

Need some help? Check out our forum.

FORUM

FYI – Having fun with datagraphs

So most of our sample code on GitHub is a result of a developer’s question that we think has general interest that other developers may find useful. This usually ends up also being a small sample application that will show the result in a simple UI.

But, sometimes we do things just for fun. For today’s example, I decided to code up the excellent descriptions on the NoServer features page

You can access the source code for the sample application here.
You can access the sample application – complete with data, images and all here.

It is interesting that now that it is so easy to create an app with a backend, that one can create the code (couple of hours), wrap it in an app (less than a couple of hours) and then blog about it.

So what this app does:

This application uses a single Objecttype (Person) and a single Collection (Persons) to demonstrate just how powerful a datagraph representation can be, and how super efficient queries can be constructed to “walk” your datagraph to get what you want with a single call in your client.

The FFDL

First, defining the backend in FFDL takes only two lines of code!

CREATE OBJECTTYPE Person (firstName STRING, lastName STRING, gender STRING, mother REFERENCE /Persons, father REFERENCE /Persons, siblings GRABBAG /Persons, picture BYTEARRAY)
CREATE COLLECTION /Persons OBJECTTYPE Person

The REFERENCE associations above (mother and father), these can be considered “one to one” relationships between person objects.

The GRABBAG association (siblings) are essentially “one to many” relationships between person objects.

The system also creates reverse relationships for all of these automatically called BackReferences that provides an extremely powerful means of “walking” your datagraph.

References

The first example shows how to retrieve an object using a REFERENCE. We first get the Person with firstName “Bart” and then get the Person referenced by the object member “father”.

var bart, homer;
ff.getObjFromUri("/Persons/(firstName eq 'Bart')/father", function(response) {
homer = response;
});
//or, if bart has already been loaded on the client
homer = bart.father;
You can see References working here.

Grab Bags

This next example shows how to retrieve a set of objects from a Grab Bag by reference. We want to get all of Bart’s aunts on his mother’s side. To do this, we will first get the Person with firstName “Bart”, then get Marge from the “mother” member Reference, then get siblings from the Grab Bag referenced by the object member “siblings”.

var grabbags;
ff.getArrayFromUri("/Persons/(firstName eq 'Bart')/mother/()/siblings/(gender eq 'Female')", function(response) {
grabbags = response;
});
You can see Grab Bags working here.

Back References

This next example shows how to retrieve objects using Back References. There are three tests here – the first gets all BackReferences to “homer”. The second gets all the Person objects from the Grab Back that reference homer by the “siblings” Reference member. The third gets all the Person objects that refer to homer by the “father” Reference member.

var homer, allBackRefs;
ff.grabBagGetAll(homer, "BackReferences", function(response) {
allBackRefs = response;
});
var homer, allSiblingBackRefs;
ff.grabBagGetAll(homer, "BackReferences.Persons.siblings", function(response) {
allSiblingBackRefs = response;
});
var homer, allFatherBackRefs;
ff.grabBagGetAll(homer, "BackReferences.Persons.father", function(response) {
allFatherBackRefs = response;
});
You can see BackReferences working here.

Queries

This last example shows how to retrieve objects using some more advanced queries. There are five examples in this section. The first example will get all the Person objects that have a “father” Reference member (Note that the returned values are deduplicated). The second is basically the same, but uses the “mother” Reference member. The third example shows the use of a logical OR to get all of Bart’s granparents. The fourth example is similar, but adds another logical OR to get all of Bart’s grandparents. The last shows a complex query to get all of Ling’s cousins (the team’s favorite query).

var fathers;
ff.getArrayFromUri("/Persons/()/father"), function(response) {
fathers = response;
});
var mothers;
ff.getArrayFromUri("/Persons/()/mother"), function(response) {
mothers = response;
});
var grandfathers;
ff.getArrayFromUri("/Persons/(firstName eq 'Bart')/father or mother/()/father"), function(response) {
grandfathers = response;
});
var grandparents;
ff.getArrayFromUri("/Persons/(firstName eq 'Bart')/father or mother/()/mother or father"), function(response) {
grandparents = response;
});
var cousins;
ff.getArrayFromUri("/Persons/(firstName eq 'Ling')/father or mother/()/siblings/()/BackReferences.Persons.mother or BackReferences.Persons.father"), function(response) {
cousins = response;
});
You can see Queries working here.

Get the Entire Datagraph

Lastly, you can actually retrieve the entire datagraph in a single query by specifying the “depth” of the leaf-level items returned. This test will fetch the entire datagraph.

var datagraph;
ff.getArrayFromUri("/Persons/()?depthRef=3&depthGb=3", function(response) {
datagraph = response;
});
You can see the entire Datagraph here.

Final note: the source code includes a couple of Server Extensions that are used to populate/unpopulate the data for this app. You may find these useful samples, although they are rather brute force and inelegant in their current form.

FYI – Retrieve an object graph in a single query with FatFractal


As a developer, I want an easy to navigate data model (a datagraph) so that I can get the data I want with fewer API calls which increases my applications performance and saves me money.

We have added a new capability that allows you to retrieve an object graph (the object as well as all of its REFERENCES and GRABBAGS) in a single API call. This can improve the the performance of your application and reduce the “chattiness” of using your API.

Previously, you could retrieve an object from a URI such as:

http://telluride.fatfractal.com/tff39/ff/resources/Showings/e-06R5_d_FfHXKHSCx-ky4

For which you would get a response like this…[toggle title="Click to see response"]

{
"result":
[
{
"ffUserCanEdit": false,
"ffUrl": "/ff/resources/Showings/e-06R5_d_FfHXKHSCx-ky4",
"qa": false,
"ffRL": "/Showings",
"updatedBy": "Os0t9Mjm7FhPemzN0uQ8U7",
"version": 1,
"updatedAt": 1346642155713,
"guid": "e-06R5_d_FfHXKHSCx-ky4",
"ffRefs":
[
{
"name": "theater",
"type": "FFO",
"url": "/ff/resources/Theaters/h3uX2Gt1j74w4yYbcifNq6"
},
{
"name": "film",
"type": "FFO",
"url": "/ff/resources/Films/KG09l8GvwRc9pUzXGMTEJ4"
}
],
"createdBy": "Os0t9Mjm7FhPemzN0uQ8U7",
"createdAt": 1346642155713,
"datetime": 1315094400000,
"clazz": "Showing"
}
],
"statusMessage":"Retrieved 1 resources from /ff/resources/Showings/e-06R5_d_FfHXKHSCx-ky4"
}
[/toggle] The response includes the object itself (in ‘result’), and includes the URIs for any references (in this case, there are two). The FatFractal SDKs would then retrieve the referenced objects (requires a minimum of two additional queries, plus even more if these objects contain references or grabbag items themselves). That makes for a lot of round trips for the queries which can be expensive and adversely impact your application performance.

Get More for Less

Now you can access the entire datagraph from the object instance at a specified depth by adding the depthRef (depth for references) and depthGb (depth for grabbags) parameters:

http://telluride.fatfractal.com/tff39/ff/resources/Showings/e-06R5_d_FfHXKHSCx-ky4?depthRef=1&depthGb=1

This will return the object AND all of the object’s REFERENCES and GRABBAGS that the user has permission to read in a single response.

The response will now look like this…[toggle title="Click to see response"]

    "result":
[
{
"ffUserCanEdit": false,
"ffUrl": "/ff/resources/Showings/e-06R5_d_FfHXKHSCx-ky4",
"qa": false,
"ffRL": "/Showings",
"updatedBy": "Os0t9Mjm7FhPemzN0uQ8U7",
"version": 1,
"updatedAt": 1346642155713,
"guid": "e-06R5_d_FfHXKHSCx-ky4",
"ffRefs":
[
{
"name": "theater",
"type": "FFO",
"url": "/ff/resources/Theaters/h3uX2Gt1j74w4yYbcifNq6"
},
{
"name": "film",
"type": "FFO",
"url": "/ff/resources/Films/KG09l8GvwRc9pUzXGMTEJ4"
}
],
"createdBy": "Os0t9Mjm7FhPemzN0uQ8U7",
"createdAt": 1346642155713,
"datetime": 1315094400000,
"clazz": "Showing"
}
],
"references":
[
{
"ffUserCanEdit": false,
"filmType": 0,
"ffUrl": "/ff/resources/Films/KG09l8GvwRc9pUzXGMTEJ4",
"director": "Ben Affleck",
"ffRL": "/Films",
"country": "U.S.",
"version": 1,
"updatedBy": "Os0t9Mjm7FhPemzN0uQ8U7",
"guid": "KG09l8GvwRc9pUzXGMTEJ4",
"updatedAt": 1346472626252,
"filmDescription": "While 50 Americans are held hostage by the Ayatollah Khomeini’s Revolutionary Guard, six U.S. Embassy employees hiding at the Canadian ambassador’s residence in Teheran stage a daring escape. “Extraction” expert Tony Mendez (Ben Affleck) enlists a veteran Hollywood make-up expert and producer (John Goodman and Alan Arkin, each giving disciplined comic performances) to create a sham sci-fi movie epic in the hopes of finding a way out of the country. Affleck directs with formidable precision and documentary-style immediacy, intercutting between turf battles between the CIA and Washington diplomats, the chaotic street violence during Iran’s revolution and the black comedy of spies pretending to be Hollywood types. This is thrilling, nail-biting entertainment.nIn person: Ben Affleck",
"duration": 120,
"title": " SNEAK: Argo",
"ffRefs": [],
"createdBy": "Os0t9Mjm7FhPemzN0uQ8U7",
"createdAt": 1346472626252,
"madePossibleBy": "",
"year": 2012,
"actors": "",
"clazz": "Film"
},
{
"ffUserCanEdit": false,
"ffUrl": "/ff/resources/Theaters/h3uX2Gt1j74w4yYbcifNq6",
"ffRL": "/Theaters",
"version": 1,
"updatedBy": "Os0t9Mjm7FhPemzN0uQ8U7",
"guid": "h3uX2Gt1j74w4yYbcifNq6",
"updatedAt": 1346359016787,
"createdBy": "Os0t9Mjm7FhPemzN0uQ8U7",
"address": "",
"createdAt": 1346359016787,
"name": "Chuck Jones Cinema",
"capacity": 500,
"longitude": -107.8478,
"latitude": 37.9361,
"sponsorString": "",
"shortName": "Chuck Jones",
"clazz": "Theater"
}
],
"statusMessage":"Retrieved 1 resources from /ff/resources/Showings/e-06R5_d_FfHXKHSCx-ky4"
}
[/toggle]

As you can see, the response includes the object itself (in ‘result’), but additionally, all objects referenced or contained in grabbags have been retrieved as well.

In your client code, using the methods provided in the various SDKs, the REFERENCE and GRABBAG items are automatically incorporated into your client object model, but now these are populated with a single request.

For more info on references, see the docs here.

Have fun!

Kevin

FYI – get all your data in one click from FatFractal


As a developer, I want to be able to export all my data (and I mean ALL my data) with one click so that I can do whatever I want to with my data, including moving to another provider – it is MY data after all.

I was asked by @objclxt a couple of days ago if he could export his data stored in FatFractal with one click. At the time, we had always generated an export of the data for customers upon request, which is fine, but not what developers want. When someone asks for functionality that I know ought to be there, I get a bit antsy, so I decided that I had put that important piece of functionality off for too long, promised Nick a 1-day turnaround, and set about it.

Twelve hours of coding and testing, eight hours of sleep, and then another 3 hours running our release candidate smoke tests later, the good news was that it worked – and so it was deployed to production.

How it works:

From the console list of your applications, you will now see a link that says “export my data”. When you click on it, you will see an alert that tells you that your data is being exported and you will receive an email with a link to a zip file with your data.

Note: the request requires authentication/ authorization for the application as does the export link in the email, which is both ephemeral and is of course an SSL link for security purposes.

So – we export a zip that contains a number of .json files (one for each of your collections with all of the object data that they contain). The object data also contains all the references to other objects, a collection for your GrabBag references, including BackReferences, and of course we also export all of your blob data in the export.

Give it a whirl and let us know what you think!

Gary

FYI – Queries that will make you smile! FatFractal’s natural language queries are very cool.


As a developer, I really want to be able to get the data back from my API in a single request so that my application performs better and I don’t get socked with excessive API requests that result from “chatty” APIs.

One of the things that developers love about FatFractal is the way that you can construct queries that can get you to what you want in a single round-trip.

First, the basics – FatFractal queries…

  1. …are written in an easy to create, easy to read, natural language style.
  2. …can be chained together to filter the results to only the results you want in a single request.
  3. …are able to “walk” your object graph in both directions.
  4. …provide the best of relational capabilities without the complexity.
  5. …are constructed as strings, not proprietary query objects.
  6. …includes a very useful random operator that for some reason, nobody else seems to offer.

Let’s say that you have movie Star, Movie and Theater objects and their respective collections (Stars, Movies and Theaters) and you want to get all the theaters that are showing movies that have a particular star acting in them within a defined distance from the user. The following explains how you can create a query string that can return that in a single request/ response to illustrate the power of nesting queries in a single construct:

/Stars/(firstName matches ‘(%3Fi)milla’ and lastName matches ‘(%3Fi)jovovich’)/BackReferences.Movies.stars/()/BackReferences.Theaters.movies/(distance (location, [57.5833, 3.8667]) lte 5000)?sort=rating desc, name asc&count=5&first=0

Creating or reading FatFractal queries goes left to right, with query constituents enclosed in parentheses with “traversals” and queries separated by forward slashes. So, breaking down the query above into pieces:

    /Stars/(firstName matches ‘(%3Fi)milla’ and lastName matches ‘(%3Fi)jovovich’)
    The first part (a query) will get the movie star(s) (Star object) from the Stars Collection with the firstName member ‘Milla’ and the lastName member ‘Jovovich’, using regular expressions that compare while ignoring case (requires proper encoding of the Java regex mode modifier ?i for ignore case to be added as %3Fi) and a boolean operator (and) to ensure a complete match. Note: we now have a set of Star objects, probably with a single object.
    /BackReferences.Movies.stars
    The second part (a traversal) will then get all the Movie objects that she starred in based on the BackReferences from the Movie objects in the Movies Collection to the star in the Stars Collection. Note: we now have a set of Movie objects.
    /()
    The third part (a query) is an empty query operator that says that will use the entire set returned by the traversal.
    /BackReferences.Theaters.movies
    The fourth part (a traversal) will then get the Theater objects from the Theaters Collection that are showing movies that Milla Jovovich starred in from the BackReferences from the Theater objects in Theaters Collection to the above set of Movie objects in the Movies Collection. Note: we now have a set of Theater objects, which is what we want.
    /(distance (location, [57.5833, 3.8667]) lte 5000)
    The fifth part (a query) will then filter that set of Theater objects by geolocation that are within 5 kilometers of the user.
    ?sort=rating desc, name asc
    The sixth part (a sort) will then sort them first by the rating member of the Theater objects in descending order, second by name member of the Theater objects in ascending order.
    &count=5&first=0
    Lastly, we then limit the result set of Theater objects to the first 5 items or less.

Summary:

So, basically we start from a Star object in the Stars Collection, walk the datagraph to Movies using a BackReference, then walk the datagraph again to Theaters using another BackReference, and then filter the set of Theater objects further based on location, order the results by both the rating and name member values and limit the number of responses to a specified maximum count. Voila – you get what you want, ordered as you want, all in one go with a single, easy to understand query – this simply cannot be done with any other BaaS provider.

Also, let me point out the random operator in the FatFractal query language as it is an example of a very common use case.

Let’s say you want to retrieve a random movie Star from the Stars collection. The query looks like this:

“/Stars/(guid eq random(guid))”

This is incredibly useful as otherwise, you would have to retrieve the entire set of Star objects from the Stars collection over the wire and then pick a random one from that set in your client code. Easy enough to program, but grossly inefficient in terms of client resources and network performance. Otherwise, you could *probably* retrieve a random item with two queries – first, by querying the collection to get a count, and if the API has an ordered ID (as many do), then you could generate a random ID in your client code and fetch that one (hoping that it has not been deleted). At any rate, getting a random object from a collection is a very common use case and I do find it bizarre that nobody else support this as easily as FatFractal does and forces you to use very inelegant client solutions.

One more point to make is that queries can be used in the context of GETS as well as within Server Extensions and Event Handlers.

All for now – have fun with your queries!

Kevin

For more information on FatFractal Query language, the docs are here.

For more information on how to write queries with FatFractal, the docs are here.

PaaS? BaaS? What?

Lots of people ask about the origins of NoServer, which is what we call FatFractal’s BaaS module.

We were in the process of building out our cloud fabric and one night, while building yet another demo app, I decided that all developer-centric cloud platforms really also ought to offer a way for developers to get 90% of what they need for the server-side of their app, out-of-the-box, while also trying to ensure that we not fence them in – i.e. give lots of functionality with minimum effort whilst also giving flexibility and extensibility to go beyond the box as you reach the edges of what is initially provided.

The 90% boiled down to the following. The cloud should:

  • Store and retrieve my objects.
  • Securely – permissions and security should be robust and at a fundamental level.
  • Efficiently – with easy to create queries that get me what I want.
  • Execute code on the backend when my data changes (events).
  • Not prevent me from doing anything else that I want to do.



The last point is really important! Whilst getting the other things makes my life a lot easier and will make me very happy for a month or two, if the flip side is that I then get stuck and as a result get frustrated and miserable, then I wouldn’t want it. So by June 2012 we had built enough to trial with customers – who loved it! – we launched our public beta in September, worked through the kinks in Oct-Dec :-) and we are now really focussed on getting the word out.

Meanwhile, some new marketing terms were emerging from other similar, but essentially different approaches. Platform as a Service (PaaS) and Backend as a Service (BaaS) are now terms that are used widely.

Upon reflection, it seems to me that they are both missing the point. PaaS is still very focused on a historical view of a set of IT problems, i.e. “stacks”. BaaS is seeking to provide more of a turnkey solution for application developers that treats the backend as a black box and is extremely limiting, invasive and prescriptive (smells like attempts at lock-in to me).

IMHO – what the market needs is a bit of both: a very smart cloud fabric that deals with infrastructure, monitoring, scalability, reliability, resilience etc, and also provides the ease of creation and out-of-the-box features that BaaS does, but does not limit the developer as a result. If I want to deploy some Java Code, Ruby Code or anything else in order to do something that I want to do, right alongside my “BaaS” stuff, then I should be able to.

So – in a nutshell, what we have so far built into NoServer is:

  • True fully qualified REST URIs
  • CRUD persistence
  • SDKs that operate at the data level and do not encroach into your code unnecessarily.
  • Amazingly straightforward modelling of your object graph’s relationships
  • Natural language, chained queries nested as deep as you want.
  • Strong authentication. SSL of course. Declarative access controls for your data which is sufficient for almost all use cases while also providing programmatic control (directly setting ACLs on objects) for when the declarative approach isn’t enough (which, so far, has been incredibly rare)
  • Exposed CRUD events with defined Event Handlers that execute your custom JavaScript code when they occur with synchronous (PRE or POST) as well as asynchronous modes. Currently we use RingoJS which allows you to do pretty much anything you need to do, including make HTTP requests; we also supply an API for sending Apple/Android push notifications, sending emails, image processing, and of course full access to your app’s datastore.
  • Server Extensions that allow you to extend your API with whatever server-side processing you want, and the ability to schedule scripts to run whenever you like



We’re pretty flexible at the server-side already, and I think you’ll agree we’re already better than (or *at least* as good as!) our BaaS competitors, but there’s lots more to come. Here’s a list of some of what’s coming over the next six months or so!

  • Greatly enhanced and pluggable authentication framework so that you can script integration with pretty much whatever 3rd-party identity / authentication cloud you like
  • Event Handlers for GET requests, and ‘virtual’ collections
  • Allowing mix and match of the powerful declarative access controls we already have with your own javascript for the really really edge cases (like, only allow access to this object if the current user lives in Idaho)
  • Enabling integration between NoServer and Rails backends. This is a big step towards completing our vision – i.e. if you reach the limits of what NoServer provides, then you should be able to extend your backend with some Ruby-on-Rails, or Java servlets, or NodeJS, or whatever
  • Allowing use of NodeJS modules in your NoServer server-side scripting

  • Full support for NodeJS applications on the FatFractal PaaS, and integration between NoServer / Node / Rails / Servlets modules in your app’s backend
  • And last, but definitely not least – and probably the most personally exciting to me given where I have spent a lot of my working life (i.e. large enterprise IT) – we will be working on evolving our products so that they can be leveraged by enterprises. More on that when the time comes :-)



I hope that you like what we have built – would very much appreciate your feedback.

Cheers,

Gary

In case you’re curious – NoServer is a tribute to the no-stuff (chamber, room, globe, ship) in Frank Herbert’s most excellent Dune series.

FYI – FatFractal supports many many to many many relationships (huh?)!


As a developer, I want to easily add many to many relationships between my objects without join tables so that I get a rich data model without getting data-model nightmares

With FatFractal, you can create amazing relationships between your objects. One of the things that makes this particularly useful is the ability to add multiple kinds of relationships from any particular object:

Example: If in your application.ffdl file that was created when you scaffolded your app, you include the following

CREATE OBJECTTYPE Comment (profile REFERENCE /Profile, comment STRING, relatedTo REFERENCE)
CREATE COLLECTION /Comment OBJECTTYPE Comment

This provides a ubiquitous “Comment” object that has two references and a “Comment” Collection typed to contain only “Comment” objects.

First, you will see a reference to the author public profile:

profile REFERENCE /Profile

This defines a relationship for the profile member which is bound to the /Profile Collection (which happens to contain only Profile objects thus “typing” the reference).

It also includes a reference called relatedTo:

relatedTo REFERENCE

This relationship is unique in that there is no Collection specifed, and thus no Objecttype implied. This means that the relationship for the relatedTo member can be to any object in any Collection of any Objecttype – i.e. is not “typed” and not bound to any particular Collection.

So now I have a comment object that has a reference to any other object in any other collection – pretty nice! All comments can are accessible via a single collection (/Comment) which is also nice.

But – what makes this truly useful is that you can also access comments via BackReferences to an object. For example, to see all comments (of Objecttype Comment) that relates to any particular object at all (as example, supR – a SupportRequest instance):

ff.getArrayFromUri(supR.ffUrl + "/BackReferences.Comment",
function(anArrayOfComments) {
// This returns an array of "Comments" that relateTo supR!!
},
function(onError) {}
}

Where the SupportRequest object model is defined as:

function SupportRequest() {
this.clazz = "SupportRequest";
this.subject = null;
this.body = null;
this.relatedTo = null;
this.profile = null;
return this;
}

Note that the SupportRequest does not even include anything about “Comment” in its definition. This means that you add relationships easily and intuitively without the pain of onerous data models – in short without the nightmares.

Hope this spurs some ideas for your application…

For more on the background of this, see Gary’s blog here.

For more details about FFDL, see the documentation here.

For more details about Queries, see the documentation here.

For more details about References and BackReferences, see the documentation here.

Kevin

Contact