Testing https://github.com/gohugoio/hugo/pull/11894

gohugoio-hugo-pull-11894-notes.md

✔️ Build status

With v0.121.2:

                   | EN  
-------------------+-----
  Pages            |  8  
  Paginator pages  |  0  
  Non-page files   |  0  
  Static files     |  1  
  Processed images |  0  
  Aliases          |  0  
  Sitemaps         |  1  
  Cleaned          |  0  

With PR #11830:

                   | EN  
-------------------+-----
  Pages            | 10  
  Paginator pages  |  0  
  Non-page files   |  0  
  Static files     |  0  
  Processed images |  0  
  Aliases          |  0  
  Cleaned          |  0  

✔️ Ctrl + C from server

On a few occassions I was not able to kill the server with Ctrl+C on the first try, and after several times it stopped with this message:

Error: context deadline exceeded

Hang after error when running server

I've a resources.GetRemote call that's currently returning a bad request (400) error, and at the moment that is expected (the provider site is having issues). I expected the hugo server to terminate after throwing the error, but I have to Ctrl+C to stop the server. With v0.121.2 the server terminates immediately. This is an intermittent problem, and I have not been able to create a mimimal failing example yet.

✔️ Issue with .Page.GetPage when called from a leaf bundle targeting a regular page; repeatable with hugo and hugo server.

git clone --single-branch -b hugo-github-issue-11830-a https://github.com/jmooring/hugo-testing hugo-github-issue-11830-a
cd hugo-github-issue-11830-a
hugo

content/
└── s1/
    ├── p1/
    │   └── index.md
    └── p2.md

single.html

{{ with .GetPage "p2" }}
  <a href="{{ .RelPermalink }}">{{ .LinkTitle }}</a>
{{ else }}
  {{ errorf "Unable to .Page.GetPage p2" }}
{{ end }}

When rendering p1 (the leaf bundle), the single page template cannot get p2.

This works as expected with v0.121.2.

Good catch, surprised that we didn't have a test case for it. Fixed in gohugoio/hugo@8f6cf63

✔️ This one seems to be related to .Page.Store and/or a code block render hook, and only occurs when running hugo server.

git clone --single-branch -b hugo-github-issue-11830-b https://github.com/jmooring/hugo-testing hugo-github-issue-11830-b
cd hugo-github-issue-11830-b
hugo server

When you start the server and visit the home page, you should see a chart rendered by chart.js... that's the way it works with v0.121.2. But instead you see nothing. While the server is running, modify the code block render hook (e.g., add a character to the bottom of the template). As soon as you save the changes, the chart appears.

This is reproducible with a Mermaid diagram using the Mermaid setup described in our documentation.

✔️ I found this in a site in the wild... happens when running hugo or hugo server.

panic: key must not end with '/': "/tags/"

content/_index.md

+++
title = 'home'
tags = ''        # note the empty string
+++

No problems with v0.121.2

I am embarrassed that it took me almost 30 minutes to find the problem.

git clone --single-branch -b hugo-github-issue-11830-c https://github.com/jmooring/hugo-testing hugo-github-issue-11830-c
cd hugo-github-issue-11830-c
hugo server

Note that this front matter doesn't cause any problems:

+++
title = 'home'
tags = []     # empty array instead of empty string
+++

✔️ Another issue with .Page.GetPage when called from a leaf bundle targeting a regular page with a path beginning with ../.

git clone --single-branch -b hugo-github-issue-11830-d https://github.com/jmooring/hugo-testing hugo-github-issue-11830-d
cd hugo-github-issue-11830-d
hugo server

content/
└── s1/
    ├── p1/
    │   └── index.md
    └── p2.md

single.html

{{/* note that path starts with ../ */}}
{{ with .GetPage "../p2" }}   
  <a href="{{ .RelPermalink }}">{{ .LinkTitle }}</a>
{{ else }}
  {{ warnf "Unable to .Page.GetPage ../p2" }}
{{ end }}

When rendering p1 (the leaf bundle), the single page template cannot get p2.

This works with v0.121.2.

---

This can be a little confusing to think about. From a file path perspective, the ../p2 path is correct. But from a logical page perspective, p1 and p2 are on the same level in the tree.

Having said that, I think the way it works in v0.121.2 is correct.

OK, thanks for the testing, much appreciated. To make this easier to reason about (for both of us), I have moved this to a branch in the main repo (not my fork):

• PR: #11894
• Branch: https://github.com/gohugoio/hugo/tree/develop2024

This one seems to be related to .Page.Store and/or a code block render hook, and only occurs when running hugo server.

I suspect this comes from some subtle change in "double rendering" of the current page when in fast render mode, which I guess masks a bug in the templates. If you run with 0.121 with regular hugo you see that the JS import is missing from the HTML file.

You need to do something ala:

{{ $content := .Content }}
{{ if .Store.Get "chart-js" }}
  <script src="https://cdn.jsdelivr.net/npm/chart.js"></script>
{{ end }}

<pre>
You should see a chart below.

If you don't, add a character to the bottom of
layouts/_default/_markup/render-codeblock-chartjs.html,
then the page will reload and you will see the chart.

With v0.121.2 you will see the chart immediately after starting huugo server.
</pre>

<div style="width: 768px">
  {{ $content }}
</div>

{{ with .GetPage "../p2" }}

OK, this one I don't understand and I have read your comment twice. I have pushed a test that matches my world view on this, and I have included both below. The first is from your first issue reported on this (which was an obvious bug), the second is my added test for your latest case:

func TestGetPageBundleToRegular(t *testing.T) {
	files := `
-- hugo.toml --
-- content/s1/p1/index.md --
---
title: p1
---
-- content/s1/p2.md --
---
title: p2
---
-- layouts/_default/single.html --
{{ with .GetPage "p2" }}
  OK: {{ .LinkTitle }}
{{ else }}
   Unable to get p2.
{{ end }}
`

	b := Test(t, files)
	b.AssertFileContent("public/s1/p1/index.html", "OK: p2")
	b.AssertFileContent("public/s1/p2/index.html", "OK: p2")
}

func TestGetPageBundleToRegular2(t *testing.T) {
	files := `
-- hugo.toml --
-- content/s1/p1/index.md --
---
title: p1
---
-- content/s1/p2.md --
---
title: p2
---
-- content/p2.md --
---
title: p2_root
---
-- layouts/_default/single.html --
../p2: {{ with .GetPage "../p2" }}{{ .Title }}{{ end }}|
`

	b := Test(t, files)
	b.AssertFileContent("public/s1/p1/index.html", "../p2: p2_root|")
	b.AssertFileContent("public/s1/p1/index.html", "../p2: p2_root|")
}

My take on this is that:

• s1/p1/index.md and s1/p2.md both are logicallly on the same level.
• So, p1 should be the relative path to p1 from both of them.
• If you do ../p2 you go one level up and arrive in the s1 section.
• If you argue that doing ../p2 from p1 should give p2 on the same level, then the first test needs to be adjusted accordingly.

EDIT: But you're right, this behaves differently in the master branch, which puzzles me a little. I don't think we had existing tests for this. To me this is a bug fix, but I need to think a little.

Also, I have pushed a fix for the tags panic: c73cb3db7932b44eacb069124b4868b29b73f2ef

OK, I have spent a few hours alpine skiing in -15 celcius which usually brings some clarity ...

I have adjusted my test case below (now it fails) to reflect how I think it should work:

• It has been important to me to be able to have portable links (GitHub), so ../p2.md should work the same as it would on GitHub (which I assume points to the folder above).
• But ../p2 should in my head navigate in Hugo's logical/canonical tree. There's several reasons for that, but the example test cases seen above makes the case I think: p2 and ../p2 cannot point to the same page.

func TestGetPageBundleToRegular2(t *testing.T) {
	files := `
-- hugo.toml --
-- content/s1/p1/index.md --
---
title: p1
---
-- content/s1/p2.md --
---
title: p2
---
-- content/p2.md --
---
title: p2_root
---
-- layouts/_default/single.html --
../p2: {{ with .GetPage "../p2" }}{{ .Title }}{{ end }}|
../p2.md: {{ with .GetPage "../p2.md" }}{{ .Title }}{{ end }}|
`

	b := Test(t, files)
	b.AssertFileContent("public/s1/p1/index.html", "../p2: p2_root|", "../p2.md: p2|")
	b.AssertFileContent("public/s1/p2/index.html", "../p2: p2_root|")
}

• Adjust it so relative .GetPage with extension is portable Hugo vs Github

OK, with gohugoio/hugo@58e8813 all other tests + the one above passes.

Related, I see that my test repo https://github.com/bep/portable-hugo-links doesn't fully work, suggesting some GetPage odditites that I'm going to look into.

✔️

I suspect this comes from some subtle change in "double rendering" of the current page when in fast render mode, which I guess masks a bug in the templates. If you run with 0.121 with regular hugo you see that the JS import is missing from the HTML file.

And you fixed the masked bug, so now you see it failing when it should be failing. That is a very good thing. I didn't notice the absence of prior content evaluation in v0.121.2 because "Hey, it's working great when I view the site locally."

👍 👍 👍

✔️ This is a terrible idea, but conceptually it helps me think about GetPage...

.GetPageByLogicalPath (tree)
.GetPageByFilePath (file system)

With one method we have to determine user intent based on something, and that's the part that makes my head hurt due to ambiguity. Ultimately, as long as there's some degree of portability, my primary goal is to be able to document the behavior. So I'm fine with your world view.

✔️ I was verifying that #7436 has been fixed (it's a multilingual, multihost site). The files are generated in the correct location, but the content of some of the files is from the wrong language.

git clone --single-branch -b hugo-github-issue-7436 https://github.com/jmooring/hugo-testing hugo-github-issue-7436
cd hugo-github-issue-7436

rm -rf resources public
hugo
printf "\n\n# public/en/posts/post-1/style.css\n"
cat           public/en/posts/post-1/style.css
printf "\n\n# public/en/posts/post-1/style.min.css\n"
cat           public/en/posts/post-1/style.min.css
printf "\n\n# public/fr/posts/post-1/style.css\n"
cat           public/fr/posts/post-1/style.css
printf "\n\n# public/fr/posts/post-1/style.min.css\n"
cat           public/fr/posts/post-1/style.min.css

Note that the related/similar #7437 is fixed by this PR:

git clone --single-branch -b hugo-github-issue-7437 https://github.com/jmooring/hugo-testing hugo-github-issue-7437
cd hugo-github-issue-7437

rm -rf resources public
hugo
printf "\n\n# public/en/posts/post-1/style.css\n"
cat           public/en/posts/post-1/style.css
printf "\n\n# public/en/posts/post-1/style.min.css\n"
cat           public/en/posts/post-1/style.min.css
printf "\n\n# public/fr/posts/post-1/style.css\n"
cat           public/fr/posts/post-1/style.css
printf "\n\n# public/fr/posts/post-1/style.min.css\n"
cat           public/fr/posts/post-1/style.min.css

✔️ Multilingual, single host site...

content/
└── posts/
    └── post-1/
        ├── a.jpg
        ├── index.de.md  <-- includes ![alt](a.jpg) in markdown
        └── index.md     <-- includes ![alt](a.jpg) in markdown

public/
├── de/
│   └── posts/
│       └── post-1/   <-- a.jpg should be here as well
│           └── index.html  
├── en/
│   └── posts/
│       └── post-1/
│           ├── a.jpg
│           └── index.html
└── index.html

Works as expected with v0.121.2.

git clone --single-branch -b hugo-github-issue-11894-a https://github.com/jmooring/hugo-testing hugo-github-issue-11894-a
cd hugo-github-issue-11894-a
rm -rf resources public && hugo && tree public

.GetPageByFilePath (file system)

I see where you're coming from, but

1. With a composite file system, it gets trickier.
2. Also, when we get other content sources (e.g. JSON), it gets even trickier.

Believe me, I have thought long and hard about this, and I think the current behaviour strikes a sensible balance.

• The paths on the form /mysection/mypost is a canonical content path (so to speak)
• Paths on the form /mysection/mypost.md and /mysection/mypost/index.md can be converted into canonical paths, but they also indicate a filename.

I have pushed an update that makes my portable links repo (link portability between Hugo<->GitHub) work again:

https://github.com/bep/portable-hugo-links

I had totally forgot about that repo, and I had no breaking tests ... Now we have tests. And the module mapping from that repo illustrates the the "a filepath is not always what you think it is" problem:

[module]
[[module.mounts]]
source = "layouts"
target = "layouts"
[[module.mounts]]
source = "README.md"
target = "content/_index.md"
[[module.mounts]]
source = "blog"
target = "content/posts"
[[module.mounts]]
source = "docs"
target = "content/mydocs"

Thanks for the test reports, much appareciated. I will look into them now.

@jmooring

• hugo-github-issue-11894-a is #11453 (which I have marked as breaking)
• hugo-github-issue-7436 I have pushed a fixed for.

✔️

hugo-github-issue-11894-a is #11453

Sorry, I should have been more clear (I described it from the wrong direction). Here's the problem:

cat public/de/posts/post-1/index.html 

Post 1 (de)

<p><img src="a.jpg" alt="kitten"></p>  <!-- broken image, src would have to point to /en/posts/post-1/a.jpg -->

But to do that we'd need to rewrite/hook markdown, which I don't understand. Or maybe I'm missing something.

#11453 seems to be about a page resource (captured with .Resources.Get), but in this case the image is just a dumb file that should be copied into the same directory as the index.html file.

But to do that we'd need to rewrite/hook markdown, which I don't understand. Or maybe I'm missing something.

• By default, we're not doing anything with the Markdown links.
• Your example shows a bundle with 2 translations and one image (in one language). The image can be reached in .Resources from both translations, but there will be only 1 link.
• This relative linking of shared bundled resources across languages have mostly been working because we have duplicated/copied the images into each language folder. This is is wasteful and it complicates the code. I think I have explained the motivation in the original issue.
• To handle the situation above, people will need to write a render hook that does .Resources.GetMatch .Destination or similar.
• We could (should?) add a default render hook, but I think, but we need to take that discussion after this lands into the main branch. See #11904

✔️ Regarding gohugoio/hugo#11453...

I have no problem with the implementation, but I mistakenly thought it was limited to de-duplicating processed images. Of all the changes in this PR, I suspect this one will generate the most noise.

To get out in front the change, I will add "portability" examples to the docs for both link and image render hooks, both of which are required for multilingual sites with resources shared between languages. The need for an image render hook is obvious, but a link render hook is also required (e.g., link to PDF that is a page resource).

Regarding existing render hooks in the wild... most of what I've seen doesn't focus on portability. Many, but not all, of the image render hooks that I've seen already capture the image as a resource, but that's not the case for the vast majority of link render hooks. Theme authors may need to update their hooks, and site authors may need to either create or override existing hooks.

✔️ Intermittent display of unprintable characters upon browser reload:

I've seen this 2 or 3 times. Hugo server is running, then I update the content page, then the browser reloads as shown above. I've experienced this with current versions of both Chrome and Firefox on Ubuntu, and with a couple of different test sites.

In the screen capture above, that is the last bit of text (markdown) in the content file. In the other instances where I saw this, I can't remember if the unprintable characters were at the end of a page, but in all cases it was only one sequence of these on a page.

This may be a red herring, but I remember (in a previous life) seeing something similar to this in a raster imaging pipeline. We had an intermittent problem where null bytes were unintentionally padding up to a 32-bit memory boundary (or something like that).

---

OK, I can reproduce this now.

git clone --single-branch -b hugo-github-issue-11453 https://github.com/jmooring/hugo-testing hugo-github-issue-11453
cd hugo-github-issue-11453
hugo server

While the server is running, edit content/_index.en.md.

Change this:

See Post 1 for test cases.

To this (remove the period):

See Post 1 for test cases

And save the file. In the browser you will see one unprintable character at the end of the line.

Also notice that other changes to the content on that page will not update the browser.

Tested with hugo v0.122.0-DEV-647097bf

In the commit message for #11894 , issues #9324 and #11671 are marked as fixed, but I can reproduce the problem with hugo v0.122.0-DEV-647097bf.

If we defer these, we need to remove them from the commit message.

✔️ # 9324 – Wrong Permalink for resource in multihost mode when baseURL in one language has a sub path

git clone --single-branch -b hugo-github-issue-9324 https://github.com/jmooring/hugo-testing hugo-github-issue-9324
cd hugo-github-issue-9324
rm -rf public && hugo && cat public/fr/p1/index.html

✔️ # 11671 – Renamed content directories not added to the watcher

git clone --single-branch -b hugo-github-issue-11671 https://github.com/jmooring/hugo-testing hugo-github-issue-11671
cd hugo-github-issue-11671
hugo server

In the browser you will see:

Title = Post 1
RelPermalink = /posts/post-1/

While the server is running, rename contents/posts to content/foo. The watcher detects the change, but the text displayed in the browser does not change.

---

EDIT If you decide to defer these, let me know and hold off on modifying the commit message. I have some additions to the list, and will provide a recommend message once these two are resolved or deferred.

At this point I have finished what I had planned to test, including every issue listed in the commit message.

✔️ Regarding gohugoio/hugo#11453...

I had a few head-scratching moments about this one this weekend. After some thinking, I decided to close/remove #7437 from the mix and introduce a new (marked as internal for now) Resource.NameOriginal method that's used as a fall back in .Resources.Get. We still normalize the resource name by removing any language code (so you can use the same name across languages), but:

1. That will not work for render hooks using .Destination
2. Also, formats without render hooks would expect to "guess" the URL using its filename.

All of this is reflected in my new test:

func TestBundledResourcesMultilingualDuplicateResourceFiles(t *testing.T) {
	t.Parallel()

	files := `
-- hugo.toml --
baseURL = "https://example.com/"
[markup]
[markup.goldmark]
duplicateResourceFiles = true
[languages]
[languages.en]
weight = 1
[languages.en.permalinks]
"/" = "/enpages/:slug/"
[languages.nn]
weight = 2
[languages.nn.permalinks]
"/" = "/nnpages/:slug/"
-- content/mybundle/index.md --
---
title: "My Bundle"
---
{{< getresource "f1.txt" >}}
{{< getresource "f2.txt" >}}
-- content/mybundle/index.nn.md --
---
title: "My Bundle NN"
---
{{< getresource "f1.txt" >}}
f2.nn.txt is the original name.
{{< getresource "f2.nn.txt" >}}
{{< getresource "f2.txt" >}}
{{< getresource "sub/f3.txt" >}}
-- content/mybundle/f1.txt --
F1 en.
-- content/mybundle/sub/f3.txt --
F1 en.
-- content/mybundle/f2.txt --
F2 en.
-- content/mybundle/f2.nn.txt --
F2 nn.
-- layouts/shortcodes/getresource.html --
{{ $r := .Page.Resources.Get (.Get 0)}}
Resource: {{ (.Get 0) }}|{{ with $r }}{{ .RelPermalink }}|{{ .Content }}|{{ else }}Not found.{{ end}}
-- layouts/_default/single.html --
{{ .Title }}|{{ .RelPermalink }}|{{ .Lang }}|{{ .Content }}|
`
	b := Test(t, files)

	// helpers.PrintFs(b.H.Fs.PublishDir, "", os.Stdout)
	b.AssertFileContent("public/nn/nnpages/my-bundle-nn/index.html", `
My Bundle NN
Resource: f1.txt|/nn/nnpages/my-bundle-nn/f1.txt|
Resource: f2.txt|/nn/nnpages/my-bundle-nn/f2.nn.txt|F2 nn.|
Resource: f2.nn.txt|/nn/nnpages/my-bundle-nn/f2.nn.txt|F2 nn.|
Resource: sub/f3.txt|/nn/nnpages/my-bundle-nn/sub/f3.txt|F1 en.|
`)

	b.AssertFileContent("public/enpages/my-bundle/f2.txt", "F2 en.")
	b.AssertFileContent("public/nn/nnpages/my-bundle-nn/f2.nn.txt", "F2 nn")

	b.AssertFileContent("public/enpages/my-bundle/index.html", `
Resource: f1.txt|/enpages/my-bundle/f1.txt|F1 en.|
Resource: f2.txt|/enpages/my-bundle/f2.txt|F2 en.|
`)
	b.AssertFileContent("public/enpages/my-bundle/f1.txt", "F1 en.")

	// Should be duplicated to the nn bundle.
	b.AssertFileContent("public/nn/nnpages/my-bundle-nn/f1.txt", "F1 en.")
}

Also, thanks for the other bug reports, I will look at them shortly.

The "non printable character" issue is fixed in gohugoio/hugo@991a8d7

9324 – Wrong Permalink for resource in multihost mode when baseURL in one language has a sub path

I have pushed a fix for this, which after the other related work in this area simplified this a lot.

After some thinking, I decided to close/remove #7437 from the mix and introduce a new (marked as internal for now) Resource.NameOriginal method that's used as a fall back in .Resources.Get.

I get it. That's exactly how I handled https://discourse.gohugo.io/t/multi-language-images-and-page-bundles/37325/8... a fallback in the render hook.

But now we don't have to do that because the fallback is baked-in. Nice.

11671 – Renamed content directories not added to the watcher

I have pushed a fixed for this (but the problem wasn't really the watcher). I'm not totally happy with how renames/new content is implemented, but it should work OK.

I recommend changing the issues referenced in the commit message for gohugoio/hugo#11894.

recommendation

Closes #11455
Closes #11549

Fixes #7425
Fixes #7436
Fixes #7544
Fixes #7882
Fixes #7960
Fixes #8255
Fixes #8307
Fixes #8863
Fixes #8927
Fixes #9192
Fixes #9324
Fixes #10169
Fixes #10364
Fixes #10482
Fixes #10630
Fixes #10656
Fixes #10694
Fixes #10918
Fixes #11439
Fixes #11453
Fixes #11457
Fixes #11466
Fixes #11540
Fixes #11551
Fixes #11556
Fixes #11654
Fixes #11661
Fixes #11663
Fixes #11664
Fixes #11669
Fixes #11671
Fixes #11807
Fixes #11808
Fixes #11809
Fixes #11815
Fixes #11840
Fixes #11853
Fixes #11860
Fixes #11883
Fixes #11904

Explanation...

Issues removed from list

#7437 - This is now closed; closed by bep on 2024-01-21.
#8498 - Disabling the taxonomy kind does not disable the term kind, and I think this is a good thing. Unless I'm missing something, we should close this separately.
#9343 - This is now closed; closed by bep on 2023-11-02.
#10104 - This is now closed; resolved in v0.93.0.
#10380 - This is now closed; duplicate of 11808.

Issues added to the list

I have reproduced these issues, and verified that they are fixed by gohugoio/hugo#11894

#7960
#8255
#8863
#10169
#10364
#10482
#10630
#10656
#10918
#11853

@jmooring I have updated the "master list" here gohugoio/hugo#11549 with your comments, I will update the commit message later. While sleeping, I seem to have a better approach to handling content renames/additions when running the server, so I will adjust that later, but other than that I'm pretty happy -- goal is to merge before weekend.

Converting this to a private gist.

Converting this to a private gist.

Disregard. The "convert to private" feature was removed a long time ago. I'll leave this here for a week or so, but at some point I will delete it.