Instantly share code, notes, and snippets.

Embed
What would you like to do?
Hugo JS Searching with Fuse.js

Client side searching for Hugo.io with Fuse.js

WHY

This gist shows how to implement client side searching with nothing but Hugo and a few common JS tools.

  • No NPM, grunt, etc
  • No additional build time steps, just hugo as you would normally.
  • Easy to swap out choice of client side search tools, anything that can use a json index
  • Highlights matching keywords in results

Sample

You can visit the Hugo Resume Theme with example site to quickly explore this feature, or visit live site (try "devops", "atlassian developer", or "rest api" as good sample searches).

Setup

  1. Add the file shown here in root directory or under themes/<themeName>
  2. Add JSON as additional output format in config.toml
  3. hugo
  4. Visit localhost:1313/search

Files

content/search.md

---
title: "Search Results"
sitemap:
  priority : 0.1
layout: "search"
---


This file exists solely to respond to /search URL with the related `search` layout template.

No content shown here is rendered, all content is based in the template layouts/page/search.html

Setting a very low sitemap priority will tell search engines this is not important content.

This implementation uses Fusejs, jquery and mark.js


## Initial setup

Search  depends on additional output content type of JSON in config.toml
\```
[outputs]
  home = ["HTML", "JSON"]
\```

## Searching additional fileds

To search additional fields defined in front matter, you must add it in 2 places.

### Edit layouts/_default/index.JSON
This exposes the values in /index.json
i.e. add `category`
\```
...
  "contents":{{ .Content | plainify | jsonify }}
  {{ if .Params.tags }},
  "tags":{{ .Params.tags | jsonify }}{{end}},
  "categories" : {{ .Params.categories | jsonify }},
...
\```

### Edit fuse.js options to Search
`static/js/search.js`
\```
keys: [
  "title",
  "contents",
  "tags",
  "categories"
]
\```


layouts/_default/search.html

This is the page rendered when viewing /search in your browser. THis example uses the template functionality of "base" and "blocks", to add my required JS files right above </body> but only on this page. You can use any template, as long as you include the 3rd part libs (jquery, fuse, mark.js) before search.js, it will work.

{{ define "footerfiles" }}
<script src="https://code.jquery.com/jquery-3.3.1.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/fuse.js/3.2.0/fuse.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/mark.js/8.11.1/jquery.mark.min.js"></script>
<script src="{{ "js/search.js" | absURL }}"></script>
{{ end }}
{{ define "main" }}
<section class="resume-section p-3 p-lg-5 d-flex flex-column">
  <div class="my-auto" >
    <form action="{{ "search" | absURL }}">
      <input id="search-query" name="s"/>
    </form>
    <div id="search-results">
     <h3>Matching pages</h3>
    </div>
  </div>
</section>
<!-- this template is sucked in by search.js and appended to the search-results div above. So editing here will adjust style -->
<script id="search-result-template" type="text/x-js-template">
    <div id="summary-${key}">
      <h4><a href="${link}">${title}</a></h4>
      <p>${snippet}</p>
      ${ isset tags }<p>Tags: ${tags}</p>${ end }
      ${ isset categories }<p>Categories: ${categories}</p>${ end }
    </div>
</script>
{{ end }}

static/js/search.js

This file uses jquery, fuse.js, mark.js to search the hugo created index, and return matching content, with highlighting.


summaryInclude=60;
var fuseOptions = {
  shouldSort: true,
  includeMatches: true,
  threshold: 0.0,
  tokenize:true,
  location: 0,
  distance: 100,
  maxPatternLength: 32,
  minMatchCharLength: 1,
  keys: [
    {name:"title",weight:0.8},
    {name:"contents",weight:0.5},
    {name:"tags",weight:0.3},
    {name:"categories",weight:0.3}
  ]
};


var searchQuery = param("s");
if(searchQuery){
  $("#search-query").val(searchQuery);
  executeSearch(searchQuery);
}else {
  $('#search-results').append("<p>Please enter a word or phrase above</p>");
}



function executeSearch(searchQuery){
  $.getJSON( "/index.json", function( data ) {
    var pages = data;
    var fuse = new Fuse(pages, fuseOptions);
    var result = fuse.search(searchQuery);
    console.log({"matches":result});
    if(result.length > 0){
      populateResults(result);
    }else{
      $('#search-results').append("<p>No matches found</p>");
    }
  });
}

function populateResults(result){
  $.each(result,function(key,value){
    var contents= value.item.contents;
    var snippet = "";
    var snippetHighlights=[];
    var tags =[];
    if( fuseOptions.tokenize ){
      snippetHighlights.push(searchQuery);
    }else{
      $.each(value.matches,function(matchKey,mvalue){
        if(mvalue.key == "tags" || mvalue.key == "categories" ){
          snippetHighlights.push(mvalue.value);
        }else if(mvalue.key == "contents"){
          start = mvalue.indices[0][0]-summaryInclude>0?mvalue.indices[0][0]-summaryInclude:0;
          end = mvalue.indices[0][1]+summaryInclude<contents.length?mvalue.indices[0][1]+summaryInclude:contents.length;
          snippet += contents.substring(start,end);
          snippetHighlights.push(mvalue.value.substring(mvalue.indices[0][0],mvalue.indices[0][1]-mvalue.indices[0][0]+1));
        }
      });
    }

    if(snippet.length<1){
      snippet += contents.substring(0,summaryInclude*2);
    }
    //pull template from hugo templarte definition
    var templateDefinition = $('#search-result-template').html();
    //replace values
    var output = render(templateDefinition,{key:key,title:value.item.title,link:value.item.permalink,tags:value.item.tags,categories:value.item.categories,snippet:snippet});
    $('#search-results').append(output);

    $.each(snippetHighlights,function(snipkey,snipvalue){
      $("#summary-"+key).mark(snipvalue);
    });

  });
}

function param(name) {
    return decodeURIComponent((location.search.split(name + '=')[1] || '').split('&')[0]).replace(/\+/g, ' ');
}

function render(templateString, data) {
  var conditionalMatches,conditionalPattern,copy;
  conditionalPattern = /\$\{\s*isset ([a-zA-Z]*) \s*\}(.*)\$\{\s*end\s*}/g;
  //since loop below depends on re.lastInxdex, we use a copy to capture any manipulations whilst inside the loop
  copy = templateString;
  while ((conditionalMatches = conditionalPattern.exec(templateString)) !== null) {
    if(data[conditionalMatches[1]]){
      //valid key, remove conditionals, leave contents.
      copy = copy.replace(conditionalMatches[0],conditionalMatches[2]);
    }else{
      //not valid, remove entire section
      copy = copy.replace(conditionalMatches[0],'');
    }
  }
  templateString = copy;
  //now any conditionals removed we can do simple substitution
  var key, find, re;
  for (key in data) {
    find = '\\$\\{\\s*' + key + '\\s*\\}';
    re = new RegExp(find, 'g');
    templateString = templateString.replace(re, data[key]);
  }
  return templateString;
}



layouts/_default/index.json

Hugo already builds indexes of all pages, we can cherry-pick which aspects should be searchable. The result is a newly created JSON index at /index.json

{{- $.Scratch.Add "index" slice -}}
{{- range .Site.RegularPages -}}
    {{- $.Scratch.Add "index" (dict "title" .Title "tags" .Params.tags "categories" .Params.categories "contents" .Plain "permalink" .Permalink) -}}
{{- end -}}
{{- $.Scratch.Get "index" | jsonify -}}

Config.toml

Add this snippet to your config file to instruct Hugo to create the index file in JSON format. (RSS and HTML are default outputs, what's important is to add JSON.

...
[outputs]
  home = ["HTML", "RSS", "JSON"]

Alternately if using a custom _index.md for home page, you can just add the output formats to front matter.


outputs:
- html
- rss
- json

See https://gohugo.io/templates/output-formats#output-formats-for-pages

@eddiewebb

This comment has been minimized.

Owner

eddiewebb commented Mar 5, 2018

Including a sample image from my personal site. Styling is totally customizable in the HUGO template files.
screen shot 2018-03-05 at 5 21 28 pm

@kaushalmodi

This comment has been minimized.

kaushalmodi commented Mar 7, 2018

Hello,

I ended up here from: https://github.com/gohugoio/hugoDocs/pull/388/files

I have zero knowledge in JS.. how do I take care of: "This file uses jquery, fuse.js, mark.js"? Sorry, nevermind, I see that here: https://gist.github.com/eddiewebb/735feb48f50f0ddd65ae5606a1cb41ae#layoutspagesearchhtml

And secondly, why not convert this to a proper blog post? and add the link to that post in the PR?

@kaushalmodi

This comment has been minimized.

kaushalmodi commented Mar 7, 2018

Also, the search results are incomplete..

Example 1: Search "hugo"

If you search for "hugo" in the index.json, it's found in a lot many posts, than the just 2 returned in the above search.

image

Example 2: Search "python"

Again, searching for "python" in the index.json shows that it's present in 3 posts (this one being the major candidate, but missing in search results).

image

But the search.js returns only 1 result:

https://search--scripter-co.netlify.com/search?s=python

image

Example 3: Search "section"

https://search--scripter-co.netlify.com/search?s=section

Above will return no results, but my last post https://search--scripter-co.netlify.com/hugo-leaf-and-branch-bundles/ has 28 occurrences of the word "section", and searching for that word in index.json shows that that word is present in 6 posts.


In summary, thank you very much for putting together this gist. This is the best search I've got working on my site so far! Hoping to having the incomplete search results issue (major) and search highlighting issue (minor) fixed.

I am wondering if undefined tags for some posts are causing the search to abort.. and result in funky highlighting and incomplete results..

@eddiewebb

This comment has been minimized.

Owner

eddiewebb commented Mar 8, 2018

@kaushalmodi - i think you've found a bug in fuse.js actually. You can visit fusejs.io and paste your index.json on the left and run the search. You'll see there that the "section" work is not matching like it should as a raw search against the json.

I've tried playing with some of the options but I am damned if I get accurate results for "section" against that payload. You should confirm yourself and submit an issue @ https://github.com/krisk/fuse/

@kaushalmodi

This comment has been minimized.

kaushalmodi commented Mar 8, 2018

@eddiewebb Thanks for the reply. That's a bummer.. wondering what's so special about my index.json. Do searches work well for your site (I haven't yet played with your index.json)? I'll open an issue on fuse.io and copy you there.

@kaushalmodi

This comment has been minimized.

kaushalmodi commented Mar 8, 2018

@eddiewebb I do not understand the Tokenize option.. but enabling that gives better results.. at least for "python".

@kaushalmodi

This comment has been minimized.

kaushalmodi commented Mar 8, 2018

@eddiewebb Enabling tokenize works great! I have now deployed this search on my blog.. searching for "section" returns 6 posts as it did when searching the raw json. Even searching for "python" returns that obvious nim vs python post that I expected.

@kaushalmodi

This comment has been minimized.

kaushalmodi commented Mar 8, 2018

@eddiewebb One last thing with searching..

At the moment, I cannot find a way to search phrases.. For example, if I try to search "leaf bundle", I get https://scripter.co/search?s=leaf+bundle .. so looks like it is literally searching for "leaf+bundle"?

Here is the index.json. Using the same on fusejs.io with the settings:

var options = {
  shouldSort: true,
  tokenize: true,
  matchAllTokens: true,
  includeMatches: true,
  threshold: 0.1,
  location: 0,
  distance: 100,
  maxPatternLength: 32,
  minMatchCharLength: 3,
  keys: [
    "title",
    "contents",
    "tags"
]
};
var fuse = new Fuse(list, options); // "list" is the item array
var result = fuse.search("leaf bundle");

I get this result:

[
  {
    "item": {
      "title": "Hugo: Leaf and Branch Bundles",
      "contents": " Hugo v0.32 introduced 
..

But I get no result when searching that on my site.

Hopefully this is something easy to fix for you, so that the "+" is not passed in the search term when I mean to pass a space character.

@kaushalmodi

This comment has been minimized.

kaushalmodi commented Mar 8, 2018

To whoever is reading this thread and has reached this point, all of the issues I mentioned above have now been fixed! Thanks @eddiewebb!

@RickCogley

This comment has been minimized.

RickCogley commented Mar 8, 2018

Just came here via the Hugo forum, and quickly tested some Japanese and CJK on the demo on http://fusejs.io/. Wow, it works with the defaults!

@RickCogley

This comment has been minimized.

RickCogley commented Mar 18, 2018

Thank you again. I got this working on my http://live.cogley.info site today. I added a one-line javascript to the search.html page, to get the cursor to focus on the search box.

<script>
  document.getElementById('search-query').focus();
</script>
@lb13

This comment has been minimized.

lb13 commented Mar 23, 2018

Just wanted to say thanks @eddiewebb - this works perfectly!

@giabaio

This comment has been minimized.

giabaio commented Mar 27, 2018

Thanks for this --- it looks very nice and impressive. I am not an expert in using js so I may be missing something trivial and obvious (and apologies for this!). But your method doesn't seem to work on my website (https://github.com/giabaio/website) --- no doubt my fault as I probably messed up some of the relevant parts.

Basically (as you can see from the live version (http://gianlucabaio.netlify.com/), the search page is created, with a search box in which you can input terms. But the search does not appear to do anything...

I think that the tags are defined in most (if not) all the pages underlying the website and the taxonomy is activated in the toml file.
I again apologise if the issue is trivial --- but I think I reached my plateau and can't seem to advance much further...

Thanks for any help you guys can give!
Gianluca

@jhabdas

This comment has been minimized.

jhabdas commented Apr 17, 2018

Thanks for this. Used it as inspiration to riff on my own implementation using Vue.

@MicZet

This comment has been minimized.

MicZet commented Jun 13, 2018

Hello I also have problem like @giabaio - unfortunately no results are given using URL/search/?s=whatever (i tried multiple ways).. I triple checked all the steps... no error in logs ... no clues... any idea what i can check ?

@jcrincon

This comment has been minimized.

jcrincon commented Jun 21, 2018

Got this working like a charm with Hugo's Tranquilipeak theme.
See demo HERE

REQUEST:
(1) Could it be posible to render tags or categories in some sort of unordered list, so we could access its CSS properties

Never mind... THIS DID the trick (search.js)

// [ ... ]
function executeSearch(searchQuery){
  $.getJSON( "/index.json", function( data ) {
    var pages = data;
    var fuse = new Fuse(pages, fuseOptions);
    var result = fuse.search(searchQuery);
    console.log({"matches":result});
    if(result.length > 0){
      populateResults(result);
    }else{
      $('#search-results').append("<div>No se encontraron resultados</div>");
    }
    // custom code begin
    var el = $('#tag-results');
    $(el).each(function(key, val) {
        var values = $(val).html().split(',');
        $(val).html('<ul id="tag-results-list">' + $.map(values, function(v) { 
          return '<li id="tag-results-item">' + v + '</li>';
        }).join('') + '</ul>');
    });
    // end of custom code
  });
}
// [ ... ]
@dschrempf

This comment has been minimized.

dschrempf commented Aug 16, 2018

Thanks for this gist!

I am having trouble when the match occurs somewhere deeper in the post because only the first 'summaryInclude' characters are shown (defaults to 60). Is there a way to include matches that are not within the first 60 characters in the results?

@ericphanson

This comment has been minimized.

ericphanson commented Aug 25, 2018

Thanks for the gist!

@dschrempf I ran into this same issue. It seems like for one thing, when tokenize is on, the search.js code will only give you that beginning snippet no matter what. Even when I comment out the else clause doing that though, I wasn't getting good results, and it turns out that the indices Fuse was giving back didn't seem right: they would give the indices of individual characters matched, but not the actual word itself (when it found both characters matched and a word). So I modified the function to just return a snippet centered around the searchQuery itself, not whatever indices Fuse reports back. This probably isn't the right solution, but here's my modified populateResults function, in case it helps:

function populateResults(result) {
  $.each(result, function (key, value) {
    var contents = value.item.contents;
    var snippet = "";
    var snippetHighlights = [];
    var tags = [];
    // CHANGED PART
    if (fuseOptions.tokenize) {
      snippetHighlights.push(searchQuery);
      // }else{
      $.each(value.matches, function (matchKey, mvalue) {
        if (mvalue.key == "tags" || mvalue.key == "categories") {
          snippetHighlights.push(mvalue.value);
        } else if (mvalue.key == "contents") {
          let ind = contents.indexOf(searchQuery)
          let start = ind - summaryInclude > 0 ? ind - summaryInclude : 0
          let end = ind + searchQuery.length + summaryInclude < contents.length ? ind + searchQuery.length + summaryInclude : contents.length
          // start = mvalue.indices[0][0]-summaryInclude>0?mvalue.indices[0][0]-summaryInclude:0;
          // end = mvalue.indices[0][1]+summaryInclude<contents.length?mvalue.indices[0][1]+summaryInclude:contents.length;
          snippet += contents.substring(start, end);
          if (ind > -1) {
            snippetHighlights.push(contents.substring(ind, ind + searchQuery.length))
          } else {
            snippetHighlights.push(mvalue.value.substring(mvalue.indices[0][0], mvalue.indices[0][1] - mvalue.indices[0][0] + 1));
          }
        // END CHANGES
        }
      });
    }
...

Also, assuming I didn't make a mistake somewhere, the gist right now has the search page reload whenever you search something, even though it's actually just passing the searchQuery to the javascript function via the ?s= parameter in the URL, then using that the javascript fires when the page loads to execute the search. So by modifying executeSearch a little,

function executeSearch(searchQuery) {
  $.getJSON("/index.json", function (data) {
    var pages = data;
    var fuse = new Fuse(pages, fuseOptions);
    var result = fuse.search(searchQuery);
    console.log({
      "matches": result
    });
    $('#search-results').html("")
    if (result.length > 0) {
      $('#search-results').append("<h3>Matching pages</h3>");
      populateResults(result);
      renderMathInElement(document.getElementById('search-results'));
    } else {
      $('#search-results').append("<p>No matches found</p>");
    }
  });
}

it clears the search results when you send a new search (I snuck in renderMathInElement(document.getElementById('search-results')); to have katex process any math in the results as well). Then the global code can be moved to a function, as in

function searchButton()
{
  searchQuery = document.getElementById("search-query").value;
  if(searchQuery){
    $("#search-query").val(searchQuery);
    executeSearch( searchQuery);
  }else {
    $('#search-results').append("<p>Please enter a word or phrase above</p>");
  }
  return false
}

That way, if you change the search.html layout to have the form call this javascript function instead of reloading the page,

 <form action="" onSubmit="return searchButton()">
        <input id="search-query" name="s"/>
    </form>

you can "instant" results.

@Ammar-K

This comment has been minimized.

Ammar-K commented Sep 2, 2018

Many thanks for the gist.
In my case, the site.coms/search/?s={somthing} is an empty page. i have checked and found the json file generated.
my site https://github.com/Ammar-K/journal

@fooqri

This comment has been minimized.

fooqri commented Oct 22, 2018

Thanks for the gist! I added a small js change to make the search more dynamic for my needs. https://gist.github.com/fooqri/a0d94853f658fb4a4bfee2ac00ab6177
example: http://rwx.io/search/?s=emacs

@ttt733

This comment has been minimized.

ttt733 commented Dec 6, 2018

Hello, first off, thanks a lot for the gist. I've been working on integrating fuse.js into our documentation this way. I had to make a few changes to the JS portion, as our site has some content (specifically backslashes within markdown code blocks) that I couldn't get hugo to parse. Now that that's working, though, I'm having a lot of trouble making it work with our theme.
You can see my attempt at it in this pull request.
I'm not sure what needs to change to cram it into the theme, since the rest of the content on our site is markdown rather than HTML. I'm thinking that I need to get the theme to recognize the HTML result as "Content", as that's what it looks like it's reading from here. But adding {{ block "Content" . }} to search.html didn't seem to work. Anyone have any other suggestions I can try? Many thanks!

Edit
As per usual, I only solve the issue immediately after giving up and asking for help. I doubt anyone will run into this specific scenario, but just in case: I was able to integrate the search results with the content for the theme I'm using by just removing search.html and changing search.md to this:

---
title: "Search Results"
sitemap:
  priority : 0.1
---

<div id="search-results"></div>
<script id="search-result-template" type="text/x-js-template">
    <div id="summary-${key}">
        <h3><a href="${link}">${title}</a></h3>
        <p>${snippet}</p>
    </div>
</script>

Not sure if this is the cleanest or most Hugo-friendly solution, but it ended up working for us. Thanks again for this great guide!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment