Symfony2: improving elasticsearch results

November 18th, 2011 14 Comments

In my previous post I looked at integrating elasticsearch into a Symfony2 app using Elastica and the FOQElasticaBundle bundle. By the end we were indexing a Site entity and performing basic searches against the index. In the post I will look at improving how we index and search the Site entities.

We can improve the indexing of the name and keywords by switching to a different analyzer. Currently we are only going to find whole word matches, for example, if we index Lime Thinking as a site name then it will be found by a search for thinking, but not think or thinks. We can change this by instead using the snowball analyzer, this is a built in analyzer which is the same as the standard analyzer but with the edition of the snowball filter which stems the tokens. This means that words are indexed as their stems, so thinking will be indexed as think. We can then find it with searches for words such as think, thinks and thinkings. I will have a more detailed look at analyzers and filters in a future post.

We just need to make a small config change to start using this analyzer for indexing:

foq_elastica:
    clients:
        default: { host: localhost, port: 9200 }
    indexes:
        bookmarks:
            client: default
            types:
                site:
                    mappings:
                        name: { analyzer: snowball }
                        keywords: { analyzer: snowball }
                    doctrine:
                        driver: orm
                        model: LimeThinking\ExampleBundle\Entity\Site
                        provider:
                        listener:
                        finder:

We need to make some further changes though to get the benefits of this. We also need to make sure that the search terms are analyzed with the same analyzer as the indexed field. If this does not happen we will only get matches if we search for the stemmed token e.g. think will find Lime Thinking but thinking will not. Our simple query does not specify which field we are searching, this means its searches the built in _all field which, unsurprisingly, contains all the fields. This means we cannot use different analyzers for searching different fields. We are going to want to add the url at some point using a different analyzer so we need to specify each field we want to search separately.

So we now need to split up our query into several parts. For this we need to use Elastica’s query builder objects. To search on a specific field we can use a Text query, so to search on the name field we use:

/**
* @Route("/sites/search/", name="site_search")
* @Method({ "head", "get" })
* @Template
*/
public function searchAction(Request $request)
{
    $finder = $this->get('foq_elastica.finder.bookmarks.site');
    $searchTerm = $request->query->get('search');
 
    $nameQuery = new \Elastica_Query_Text();
    $nameQuery->setFieldQuery('name', $searchTerm);
 
    $sites = $finder->find($nameQuery);
    return array('sites' => $sites);
}

Notice that we pass the query object into the same method on the finder as before, this method accepts both simple search strings as well as queries built through objects. According to the elasticsearch documentation the analyzer will default to the field specific analyzer or the default one, to me this suggests that the above query will automatically use the analyzer set for the field. However this does not work for me, fortunately it easy to specify the analyzer to use for the field:

/**
* @Route("/sites/search/", name="site_search")
* @Method({ "head", "get" })
* @Template
*/
public function searchAction(Request $request)
{
    $finder = $this->get('foq_elastica.finder.bookmarks.site');
    $searchTerm = $request->query->get('search');
 
    $nameQuery = new \Elastica_Query_Text();
    $nameQuery->setFieldQuery('name', $searchTerm);
    $nameQuery->setFieldParam('name', 'analyzer', 'snowball');
 
    $sites = $finder->find($nameQuery);
    return array('sites' => $sites);
}

Our current query will of course only search the name field, what we want to do is search the name field and the keywords field using the snowball analyzer. This is done by creating another query as above for the keywords field and then using a boolean query to combine the two individual queries into one query:

/**
* @Route("/sites/search/", name="site_search")
* @Method({ "head", "get" })
* @Template
*/
public function searchAction(Request $request)
{
    $finder = $this->get('foq_elastica.finder.bookmarks.site');
    $searchTerm = $request->query->get('search');
 
    $nameQuery = new \Elastica_Query_Text();
    $nameQuery->setFieldQuery('name', $searchTerm);
    $nameQuery->setFieldParam('name', 'analyzer', 'snowball');
 
    $keywordsQuery = new \Elastica_Query_Text();
    $keywordsQuery->setFieldQuery('keywords', $searchTerm);
    $keywordsQuery->setFieldParam('keywords', 'analyzer', 'snowball');
 
    $boolQuery = new \Elastica_Query_Bool();
    $boolQuery->addShould($nameQuery);
    $boolQuery->addShould($keywordsQuery);
 
    $sites = $finder->find($boolQuery);
    return array('sites' => $sites);
}

Whilst this looks complicate each constituent part is simple and this is a good way to build more complicated queries.

A really helpful recent inclusion to the bundle is logging to the web profiler toolbar so you can see the parsed JSON query that is sent to elasticsearch. The combined query from above looks like this:

Method: GET
{ query: { bool: { should: [{ text: { name: { query: thinking, analyzer: snowball } } }, { text: { keywords: { query: thinking, analyzer: snowball } } }] } } }
Time: 8.19 ms

We have seen Text query and Boolean query here, these are just a few of the available query types. There is more information on each in the elasticsearch documentation. There is little in the way of documentation for the Elastica objects for creating these query types but the test suite provides quite a lot of example of putting them to use.

Related posts:

  1. Symfony2: Integrating elasticsearch
  2. Symfony2: First Site
Posted in PHP | Tags: , , , ,
«
»
You can leave a response, or trackback from your own site.

14 Responses to “Symfony2: improving elasticsearch results”

  1. Marcus Stöhr says:
    November 18, 2011 at 1:10 pm

    Hi Richard,

    thanks for this posting. It showed me some configuration options I wasn’t aware of but one question still remains: How do I add filters to my index? I have the following use case which I want to make work:

    Entity with property ‘title’ can contain any text, e.g. Varèse Sarabande (notice the accent). For the search I want the ability to enter ‘varese’ which should match the prior text but doesn’t. If I use ‘var*’ it finds it obviously.

    Any hints?

    - Marcus

    Reply
  2. Lukas says:
    November 18, 2011 at 5:00 pm

    What is this I see in your code .. $this->get(..)!!! :)

    Reply
  3. Menno says:
    December 14, 2011 at 3:26 pm

    Hi Richard,

    Thanks for your usefull posts on integrating elasticsearch and sf2.
    Since you seem to be an expert on the subject, maybe you can help me with the following…

    I want to search related doctrine entities with elasticsearch. I’ve found some things on nested objects, queries and filters on the elasticsearch website, but I’m not sure this can be used for searching doctrine entities with the FOQElasticaBundle.

    use case:
    Recipe
    - Cook (entity)
    - Ingredients (collection of ingredient entities)

    Search the recipe on its title and preperation description, but also search cook name and ingredient names….

    Any idea how I can achieve this?

    Thanks in advance,

    Menno.

    Reply
    • miller says:
      December 14, 2011 at 5:42 pm

      @Menno If you are only looking to return Recipe entities as the results of the search then rather than a complicated mapping in elasticsearch you could just index recipes and flatten the other fields into them e.g.

      recipe:
      mappings:
      title
      //—
      cookName
      ingredients

      Then you could either create a custom class for transforming the entities to match this or you could add methods to the Recipe entity that return the relevant data. For example if you added:

      getCookName()
      {
      return $this->getCook()->getName();
      }

      then you would automatically get the cook’s name indexed. For ingredients you could do something similar by adding a getIngredients() method in which you loop through the ingredients joining them into a string which would then get indexed.

      Reply
  4. Menno says:
    December 23, 2011 at 2:39 pm

    Hi Richard,

    Thanx for that, works like a charm.

    Yet another issue: I use the Doctrine listener, but it does not seem to work when I unpublish an item. When I publish it, it works fine, the item shows up in the result. But when I unpublish it, it keeps appearing in the result unless I update the index using the app/console.
    … any idea how I can debug to see when the index is updated?

    Reply
    • miller says:
      December 23, 2011 at 3:46 pm

      The queries sent to elasticsearch can be seen using the web debug toolbar so you should be able to see if the delete query gets sent using it. If you are redirecting after the delete is carried out then you will need to set intercept_redirects to true in your config (there is a setting for it in config_dev.yml in the standard distribution)

      Reply
      • Menno says:
        December 23, 2011 at 3:51 pm

        Well, the query is executed just fine. It’s more that the elasticsearch index is not updated when I unpublish an item (setIsPublished(false)).
        Somehow the index does get updated when I re-publish the item. In both cases the db entree is updated, so the lifetime callback is executed.
        And when I use the console to re-index elasticsearch, the result is fine again. This leads me to believe that somehow the call to re-index is not executed. And that’s what I want to debug…

        Reply
        • miller says:
          December 23, 2011 at 3:59 pm

          The toolbar should show all the queries that are executed, as it is being executed but not having the desired effect on the index then you may want to have a look at query elasticsearch directly rather than through Elastica. You can get details of how to do this using curl at the cli from the elasticsearch docs http://www.elasticsearch.org/

          Reply
  5. kell says:
    February 24, 2012 at 4:25 pm

    base.html.twig
    ———————————————————————-

    ———————————————————————
    SearchController.php

    get(‘foq_elastica.finder.website.cities’);
    $searchTerm = $request->query->get(‘search’);

    $titleQuery = new \Elastica_Query_Text();
    $titleQuery->setFieldQuery(‘title’, $searchTerm);

    $cities = $finder->find($titleQuery);
    return array(‘cities’ => $cities);
    }
    }
    ———————————————————————
    config.php

    foq_elastica:
    clients:
    default: { host: localhost, port: 80 }
    indexes:
    website:
    client: default
    types:
    cities:
    mappings:
    title: { analyzer: snowball }
    persistence:
    driver: orm
    model: XXXX\ResellerBundle\Entity\Cities
    provider:
    listener:
    finder:
    ————————————————————————

    thanks for that, i have a problem, when i am searhing I get this error:

    Notice: Undefined index: hits in /var/www/XXXX/vendor/elastica/lib/Elastica/ResultSet.php line 36

    what is that means?

    Reply
  6. Marco says:
    February 16, 2013 at 11:40 am

    Hi !
    I have a little problem.

    I want join results with related entity , and order this results for one field. The problem is that field isn’t indexed by elastica.

    So, i find a Model Transformer object but i don’t understand how i can ovveride this object or extend a transformer query object for add my custom criteria.

    Someone can help me ?! :D

    Tnx .

    Reply
  7. channa ly says:
    April 7, 2013 at 10:03 am

    Very useful article. If I want all the results highlighted, how to get this done with elasticabundle ?

    thanks in advance.

    Reply

Leave a Reply

Please use [code] and [/code] around any source code/html/xml you wish to include in a comment.

Subscribe to RSS Feed Follow me on Twitter!