Skip to content

Instantly share code, notes, and snippets.

@HugoDF
Last active October 23, 2023 09:47
Show Gist options
  • Star 4 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
  • Save HugoDF/aac2e529f79cf90d2050d7183571684b to your computer and use it in GitHub Desktop.
Save HugoDF/aac2e529f79cf90d2050d7183571684b to your computer and use it in GitHub Desktop.
Integrate lunrjs with a Hugo (gohugo.io) site.
const fs = require('fs').promises;
const {promisify} = require('util');
const frontMatterParser = require('parser-front-matter');
const parse = promisify(frontMatterParser.parse.bind(frontMatterParser));
async function loadPostsWithFrontMatter(postsDirectoryPath) {
const postNames = await fs.readdir(postsDirectoryPath);
const posts = await Promise.all(
postNames.map(async fileName => {
const fileContent = await fs.readFile(
`${postsDirectoryPath}/${fileName}`,
'utf8'
);
const {content, data} = await parse(fileContent);
return {
content: content.slice(0, 3000),
...data
};
})
);
return posts;
}
const lunrjs = require('lunr');
function makeIndex(posts) {
return lunrjs(function() {
this.ref('title');
this.field('title');
this.field('content');
this.field('tags');
posts.forEach(p => {
this.add(p);
});
});
}
async function run() {
const posts = await loadPostsWithFrontMatter(`${__dirname}/content/post`);
const index = makeIndex(posts);
console.log(JSON.stringify(index));
}
run()
.then(() => process.exit(0))
.catch(error => {
console.error(error.stack);
process.exit(1);
});
node ./build-lunrjs-index.js > static/search-index.json

The MIT License (MIT) Copyright (c) 2019-2020 Hugo Di Francesco

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

{
"devDependencies": {
"lunr": "^2.3.6",
"parser-front-matter": "^1.6.4"
},
"license": "MIT"
}
<form method="get" action="">
<input id="search" name="q" type="text" />
<button type="submit" class="button">Search</button>
<a href="/search">Clear</a>
</form>
<div id="#app"></div>
<script src="https://unpkg.com/lunr/lunr.js"></script>
<!-- Generate a list of posts so we can display them -->
{{ $p := slice }}
{{ range (where .Site.RegularPages "Section" "==" "post") }}
{{ $post := dict "link" .RelPermalink "title" .Title "content" (substr .Plain 0 200) -}}
{{ $p = $p | append $post -}}
{{ end }}
<script>
const posts = JSON.parse(
{{ $p | jsonify }}
);
const query = new URLSearchParams(window.location.search);
const searchString = query.get('q');
document.querySelector('#search').value = searchString;
const $target = document.querySelector('#app');
// Our index uses title as a reference
const postsByTitle = posts.reduce((acc, curr) => {
acc[curr.title] = curr;
return acc;
}, {});
fetch('/gen/search-index.json').then(function (res) {
return res.json();
}).then(function (data) {
const index = lunr.Index.load(data);
const matches = index.search(searchString);
const matchPosts = [];
matches.forEach((m) => {
matchPosts.push(postsByTitle[m.ref]);
});
if (matchPosts.length > 0) {
$target.innerHTML = matchPosts.map(p => {
return `<div>
<h3><a href="${p.link}">${p.title}</a></h3>
<p>${p.content}...</p>
</div>`;
}).join('');
} else {
$target.innerHTML = `<div>No search results found</div>`;
}
});
@HugoDF
Copy link
Author

HugoDF commented Aug 8, 2019

@psharma04 the HTML only works as a Hugo template

@oonid
Copy link

oonid commented Aug 11, 2019

@HugoDF do you have any example how to integrate search.html to Hugo template? or may be another article at codewithhugo? thank you.

@HugoDF
Copy link
Author

HugoDF commented Aug 11, 2019

The post is at https://codewithhugo.com/hugo-lunrjs-search-index/

The build-lunrjs-index.js script needs to be built before the corresponding Hugo template is used to generate the site.

@exprez135
Copy link

exprez135 commented Nov 7, 2019

I'm trying to implement this in a site, but am getting no search results. Firefox's Console shows this error each time I attempt to search: "TypeError: $target is null".

Edit: Also, if there are any folders in the content/post/ directory (e.g. if I have subdirectories to split by year and month), the index script ends with the error "Error: EISDIR: illegal operation on a directory, read". Any way to recursively index all directories?

@HugoDF
Copy link
Author

HugoDF commented Nov 7, 2019

@exprez135 for recursive search, you'll have to implement that, get all the paths and then load all the posts into memory, I would change https://gist.github.com/HugoDF/aac2e529f79cf90d2050d7183571684b#file-build-lunrjs-index-js-L8-L9 onwards

The only situation where $target would be null is if you don't have <div id="app"></div> somewhere in your HTML template https://gist.github.com/HugoDF/aac2e529f79cf90d2050d7183571684b#file-search-html-L6

@exprez135
Copy link

By the way, I successfully implemented that recursive search in my project using readdirp in place of readdir and some other work-arounds. You can find it at https://git.sr.ht/~exprez135/taliaferro/tree/master/scripts/build-lunrjs-index.js. My search template is at https://git.sr.ht/~exprez135/mediumish-taliaferro/tree/master/layouts/search-page/search.html.

And I just realized that I used your code without a license. Is there any way you could specify a license for people to use or declare it in the public domain?

Thank you for writing this!

@HugoDF
Copy link
Author

HugoDF commented Apr 17, 2020

@exprez135 added MIT license + set it as the license in package.json 😄 thanks for the nudge I'm glad it helped you.

@ttgiang
Copy link

ttgiang commented Jul 14, 2020

Appreciate the code. Learning a lot from it. Would you mind sharing a sample of the data file. I'm new to this and am trying to make it work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment