This post shows how you can use a RavenDB map index as a persistent view of your documents.

There was an interesting question on StackOverflow and also another one on the mailing-list (btw, did I tell you that mailing-lists suck? go and use SO instead! ) regarding how to query on a simple map index and use the index itself as the result of the query.

Ok, first we need to answer this question:

What is the difference between a map and a map/reduce index?

The obvious difference is that a map index has only one or more map functions to extract index fields out of your documents, while a map/reduce also has a reduce function that runs over the results of the map functions in a second step. Ayende has written a good explanation of this concept. This happens at indexing time, but there is also a big difference at query time:

When you query a map/reduce index, the index itself is the result of your query, whereas querying a map index will give you the original documents!

Internally, it works like this: At indexing time, RavenDB extracts information out of your input documents and uses this information to create fields on lucene documents, that make up your index (leaving tokenization and analysis aside). You need those fields, so that you can query upon them. Along with the fields that you define as the result of your map function, RavenDB will also store a field __document_id that is the key of your original document. So when you query, RavenDB uses this field to fetch the original documents and answer your query. This is the way map indexes work – it is very important to understand that concept if you want to do more advanced stuff.

Remember, we are only talking about map indexes here, because for map/reduce indexes we already get what we want. So the real question is:

How can we tell RavenDB not to load the original documents and return them as the result of our query, but instead give us the fields stored in the index as a result?

Here is the answer:

var results = documentSession.Query<Post, AggregationIndex>()
                        .AsProjection<AggregationIndex.ReduceResult>()
                        .ToList();

It’s as simple as that. Don’t be confused by the name of the inner class – there is no reduce function, this naming is just because I call every result of an index a ReduceResult. (I don’t know who came up with this convention anyway, but I’m sure I don’t like it because it produces some much confusion. I would rather call it just Result, but I’d like to be consistent with other peoples code…)

Here is the index itself:

public class AggregationIndex : AbstractIndexCreationTask<Post, AggregationIndex.ReduceResult>
{
    public class ReduceResult
    {
        public string Id { get; set; }
        public string Title { get; set; }
        public string Text { get; set; }
        public int CommentsCount { get; set; }
    }

    public AggregationIndex()
    {
        Map = posts => from post in posts
                       select new
                       {
                           post.Title,
                           post.Text,
                           CommentsCount = post.Comments.Count
                       };

        Store(x => x.Title, FieldStorage.Yes);
        Store(x => x.Text, FieldStorage.Yes);
        Store(x => x.CommentsCount, FieldStorage.Yes);
    }
}

Nothing special here, right?
Stop. Make notice of the calls to Store(). Now re-read the bold question above, where it says…

…give us the fields stored in the index…

By default, RavenDB creates the fields in the lucene documents in a way that they can be used in queries, but cannot be retrieved from the query result. So, if we want to use the index itself as a query result, we need to enable field storage.

The only field that we don’t need to set field storage explicitly on, is the document key (the Id property), because this is already done by RavenDB.

In our example above, we chose to have all our indexed fields to be stored, which is generally a not so good idea if you have large fields (like the text in a blog post) because lucene isn’t intended to be a database on its own. So instead of storing all fields it is better only to store the fields we really need and have a projection class for that. Let’s modify our code to take that into account.

public class AggregationIndex : AbstractIndexCreationTask<Post, AggregationIndex.ReduceResult>
{
    public class ReduceResult
    {
        public string Id { get; set; }
        public string Title { get; set; }
        public string Text { get; set; }
        public int CommentsCount { get; set; }
    }

    public class ResultProjection
    {
        public string Id { get; set; }
        public string Title { get; set; }
        public string CommentsCount { get; set; }
    }

    public AggregationIndex()
    {
        Map = posts => from post in posts
                       select new
                       {
                           post.Title,
                           post.Text,
                           CommentsCount = post.Comments.Count
                       };

        Store(x => x.Title, FieldStorage.Yes);
        Store(x => x.CommentsCount, FieldStorage.Yes);
    }
}

And the query…

var results = documentSession.Query<Post, AggregationIndex>()
                        .AsProjection<AggregationIndex.ResultProjection>()
                        .ToList();

It is better, because it keeps the lucene index smaller and thus makes queries even faster. Note that all the computation happens at indexing time and we only get the results at query time.

I don’t know how if there is an equivalent concept in SqlServer, but in Oracle there are materialized views that give you similar query characteristics.

As an extra point, I’d like to mention that if you are using this approach, you can count yourself to the cool CQRS kidz. I’ll leave it up to you to find out why.

In this post I’ll show you how we built a nice search using RavenDBs multi maps indexes and facets feature. But first, here is the context:

In one of our recent projects using RavenDB, we created a software system for the federal government of Upper Austria, that is intended to be a platform for all kinds of musical works and artists, both classical and contemporary. It consists of different parts, one of them is a public web application that will be used to publish news and upcoming events, but also provides a nice search feature for authors and opuses, that are stored in the database.

It is now running in test environment, but if you’re not afraid of German language, you can already try it out here: www.diemusiksammlung.at/search

This is a screenshot:

Given a single search term, the results will contain both authors and opuses, sorted by their relevance. In addition, you can drill-down the search results using facets. By clicking on the gray arrow, you can expand each facet block to see more options available. In any case, you will only see facets that return at least one document, either an opus or an author.

Ok, so what is interesting here?

Besides this facet thing (that could be the topic of another post), we have used a multi maps index to search over authors and opuses at the same time.

Here is a simplified model:

public class Opus
{
    public string Id { get; set; }
    public string Name { get; set; }
    public string SubName { get; set; }
    public DateTime? PublishingDate { get; set; }
    public List<string> Genres { get; set; }
    public string Description { get; set; }
    public List<AuthorReference> Authors { get; set; }
}

public class Author
{
    public string Id { get; set; }
    public string FirstName { get; set; }
    public string LastName { get; set; }
    public string Email { get; set; }
    public List<string> SpheresOfAction { get; set; }
    public string Biography { get; set; }

    public DateTime? DateOfBirth { get; set; }
}

public class AuthorReference
{
    public string Id { get; set; }
    public string FullName { get; set; }
}

Now, let’s imagine we have thousands of opuses and authors in our database. We want to search on those documents by a search term and expect results sorted by their relevance with full paging capabilities, no matter if they are opuses or authors.

To do that, we need an index that covers both document types. This is what multi map indexes are for. The only difference to a simple map index is, that there are at least two map functions that extract data out of your documents to populate the lucene index.

In our case, the most trivial index would probably be this one:

public class AuthorsAndOpuses : AbstractMultiMapIndexCreationTask<AuthorsAndOpuses.ReduceResult>
{
    public class ReduceResult
    {
        public string Name { get; set; }
    }

    public AuthorsAndOpuses()
    {
        AddMap<Opus>(opuses => from opus in opuses
                              select new { Name = opus.Name });

        AddMap<Author>(authors => from author in authors
                                  select new { Name = author.FirstName + " " + author.LastName });

        Index(x => x.Name, FieldIndexing.Analyzed);
    }
}

Using this index, we can query on the name property of both document types:

var results = session.Query<AuthorsAndOpuses.ReduceResult, AuthorsAndOpuses>()
                        .Where(x => x.Name.StartsWith("Dan"))
                        .As<dynamic>()
                        .ToList();

Each result item is either of type Author or Opus and you can cast them as needed. However, we cannot query on any properties that are specific to authors or opuses. If we want to do that, you need to add those fields to the index:

public class AuthorsAndOpuses : AbstractMultiMapIndexCreationTask<AuthorsAndOpuses.ReduceResult>
{
    public class ReduceResult
    {
        public string Name { get; set; }
        public string AllText { get; set; }

        public DateTime? Opus_PublishingDate { get; set; }
        public List<string> Opus_Genres { get; set; }

        public string Author_Biography { get; set; }
        public DateTime? Author_DateOfBirth { get; set; }
    }

    public AuthorsAndOpuses()
    {
        AddMap<Opus>(opuses => from opus in opuses
                               select new
                               {
                                   Name = opus.Name.Boost(3),
                                   AllText = new[]
                                                 {
                                                     string.Join(" ", opus.Authors.Select(author => author.FullName)),
                                                     opus.SubName,
                                                     opus.Description
                                                 },

                                   Opus_PublishingDate = opus.PublishingDate,
                                   Opus_Genres = opus.Genres,

                                   Author_Biography = (string)null,
                                   Author_DateOfBirth = (object)null
                               });

        AddMap<Author>(authors => from author in authors
                                  select new
                                  {
                                      Name = (author.FirstName + " " + author.LastName).Boost(3),
                                      AllText = new[]
                                                    {
                                                        author.Email,
                                                        author.Biography
                                                    },

                                      Opus_PublishingDate = (object)null,
                                      Opus_Genres = (object)null,

                                      Author_Biography = author.Biography,
                                      Author_DateOfBirth = author.DateOfBirth
                                  });

        Index(x => x.Name, FieldIndexing.Analyzed);
        Index(x => x.AllText, FieldIndexing.Analyzed);
    }
}

This index enables us to find all authors and opuses that contain a word beginning with “Dan”, no matter if the word is inside the authors biography/email or if it is part of the opus’ description.

string searchTerm = "Dan";
var results = session.Query<AuthorsAndOpuses.ReduceResult, AuthorsAndOpuses>()
    .Where(x => x.Name.StartsWith(searchTerm) || x.AllText.StartsWith(searchTerm))
    .As<dynamic>()
    .ToList();

Let’s assume we have an author whose name is “Daniel” and that author has 3 opuses. Using the query above, we will get all 4 documents. A very nice thing is, that because we have boosted the Name field in the index, the author will be the first item in the search result, followed by the 3 opuses which only contain the search term in their field AllText, which was not boosted at all.

Also note that we have included opus and author specific properties in the index above and we had to null them out in the map-functions where we didn’t have the information available. Now we can use the same index to drill down the search result using conditions that are more specific to Opus or Author. In our own application we have done that using a SearchQueryBuilder that dynamically builds the query depending on the input viewmodel that gets posted to the controller.

Using this approach it is very easy to integrate rich search capabilities into your applications. Again, because RavenDB uses Lucene under the hood and has the indexes pre-built, you can expect to get results very fast. Just try it out on our site and let me know what you think.

I just got an email from Robert Muehsig asking me for some advice on how to do searching on text fields in RavenDB. This seems to be a question that pops up quite frequently, so here are my two cents.

Given this scenario:

class User
{
    public string Id { get; set; }
    public string Name { get; set; }
}

session.Store(new User { Name = "Daniel Lang" });
session.Store(new User { Name = "Daniel Smith"});
session.SaveChanges();

Let’s see what we get when we fire some basic queries on them:

session.Query<User>().Where(x => x.Name == "Daniel Lang").ToList(); // returns just me
session.Query<User>().Where(x => x.Name.StartsWith("Daniel")).ToList(); // returns "Daniel Lang" and "Daniel Smith"
session.Query<User>().Where(x => x.Name.Contains("Daniel")).ToList(); // ATTENTION - 0 results
session.Query<User>().Where(x => x.Name.Contains("Lang")).ToList(); // also 0 results!

While the first two queries work as expected, the third and the fourth both return 0 results although they should return 2 and 1 users respectively. What has happened here?

The reason for this has to do with the way RavenDB uses Lucene to build the index here.

Some background about indexes

If you send a query to RavenDB, this is what happens on the server: RavenDB will not go and scan all your documents to check whether a document meets your query-condition or not! This would be very slow in fact. Instead, it executes the query on an index that has been created asynchronously in the background. RavenDB uses Lucene.NET to create, maintain and query those indexes. Because lucene is incredibly fast for queries and the whole indexing happens asynchronously in a background-thread on the server, both queries and writes are lightning fast with RavenDB. If you send a query for which no static index has been created before, RavenDB will create a temporary one for you automatically. This temporary index extracts (say maps) all those properties out of your documents that are needed to answer the question. Alright, I guess that’s the story you already know if you’re using RavenDB, but what does that mean for us?

The essential thing is – every query will be translated into a lucene-query in the end.

So when you write this line of code…

session.Query<User>().Where(x => x.StartsWith("Daniel")).ToList();

it will directly translate into this lucene query Name: Daniel*
The syntax is actually quite easy to understand. As you might guess, the asterisk is a placeholder/wildcard for “whatever follows” and thus represents the obvious implementation of “starts with”.

Lucenes query syntax is very powerful and allows you to do a lot of interesting things, take a look at the documentation here: Lucene query syntax.

So the obvious (and perfectly working) choice to implement string.Contains would be to use *search-term*, right?
Ok, but…

Why (the hell) does string.Contains() not work for text-search in RavenDB?

The short answer: because of ravens “safe-by-design” paradigm.

The long answer is that a leading wildcard forces lucene to do a full scan on the index and thus can drastically slow down query-performance. Lucene internally stores its indexes (actually the terms of string-fields) sorted alphabetically and “reads” from left to right. That’s the reason why it is fast to do a search for a trailing wildcard and slow for a leading one.

Because of that, string.Contains is implemented the same way as string.Equals in RavenDBs linq provider, for good reasons as we’ve seen.

How to do it right?

While you certainly can ignore RavenDBs built-in protection using this line of code

session.Advanced.LuceneQuery<User>().Where("Name: *Daniel*").ToList();
// naive implementation of string.Contains

… I don’t recommend to do so, as there is a much better alternative available.

Instead of just showing you the code, I want you to understand the concept behind it. In the code samples above, we’ve seen that we get the correct result if the whole string matches the name of a user (not case-sensitive). This has to do with the way lucene stores the strings that come in from the documents. Remember, an index in RavenDB (no matter if it was created manually or automatically) just extracts fields (properties) out of documents inside the database and passes them on to lucene.net, so that lucene stores them for later querying. When a string/text field comes in into lucene, it will either be stored as it is, or, will be analyzed before. Analyzing means that the string might be split up (say tokenized) into multiple parts (say terms) before they get stored into the index. This is very important to understand!

Think about a lucene index as a flat table with two columns, where the first column contains all the terms (whole string-properties out of documents or tokenized parts) and the second column the primary keys (the ids) of the corresponding original documents, which can later be used to load them out of RavenDBs underlying storage engine. This model is actually far away from how lucene.net really works, but I think it works quite good to explain the concept…

Ok, so you already guess that it’s the analyzing step in the indexing process, that makes the difference. You were right, by default RavenDBs uses a custom analyzer (one that doesn’t ship with lucene.net) that’s called LowerCaseKeywordAnalyzer. That means, if you don’t explicitly say you want to have another one on each of your indexed properties, this guy will be used for all of them. What it does is, it lowercases all strings before they come into lucene (thus queries are not case-sensitive). That’s fine in most cases, but if we want to do some text-searching, we will want to use an Analyzer that tokenizes every word. In our sample above, we don’t want the name “Daniel Lang” to be stored as a whole inside lucenes “table”, but instead want two rows “Daniel” and “Lang” in the first column.

The good news is, it’s quite simple to do that. Lucene ships with a handful of Analyzers and they are all available through RavenDB as well, take a look here: How indexes work

The one you want in case of our Name property is the StandardAnalyzer – it breaks words at whitespace characters and ignores common English stop-words.

Finally, here is the code:

documentStore.DatabaseCommands.PutIndex("UsersByName", new IndexDefinitionBuilder<User>
{
    Map = users => from user in users
                   select new { user.Name },
    Indexes =
        {
            { x => x.Name, FieldIndexing.Analyzed}
        }
});

Because we have set property the Name to be Analyzed, RavenDB will automatically choose the StandardAnalyzer for it. If you want to explicitly set another one, it looks quite similar (but I suggest to stick with the StandardAnalyzer in this case):

documentStore.DatabaseCommands.PutIndex("UsersByName", new IndexDefinitionBuilder<User>
{
    Map = users => from user in users
                   select new { user.Name },
    Analyzers =
        {
            { x => x.Name, "SimpleAnalyzer"}
        }
});

As soon as we have the index set-up correctly, we can now do the following query:

session.Query<User>("UsersByName").Where(x => x.Name.StartsWith("Lang")).ToList();
// will return the user "Daniel Lang", even though this string doesn't start with "Lang"

That’s all about it. As soon as you’ve got the indexing thing clear in your mind, you will love the amazing power lucene and it’s query syntax give you. Good luck!

Edit: Please note ..
…as Steve pointed out in the comment below, this doesn’t answer the question how to implement string.Contains. I’m sorry that I haven’t been very clear on that. Here’s the deal: you don’t need it! Think about it – in almost every case you actually want to have string.StartsWith on every word, because the results will be more relevant. I we had a full string.Contains, a search for “La” would not only return “Daniel Lang”, but also a user “Angela Merkel”.

However, if you really have a case where you want to do string.Contains (I cannot think of an example scenario for that right now) you can certainly do that using the leading wildcard.

One more thing: An imlementation of string.EndsWith can be done very easy and effective. All you need to do is to select a lucene analyzer that reverses the string (because analyzers are also applied for the search-term that is just as easy as that).