Debugging rssCloud on WordPress.com

README.md

I have a test application running at test.rsscloud.io on ports 80 and 9876.

The following code works:

curl --location --request POST 'https://brokenriverbooks.com/?rsscloud=notify' \
--header 'Content-Type: application/x-www-form-urlencoded' \
--data-urlencode 'domain=test.rsscloud.io' \
--data-urlencode 'port=80' \
--data-urlencode 'path=/feedupdated-s8759' \
--data-urlencode 'registerProcedure=' \
--data-urlencode 'protocol=http-post' \
--data-urlencode 'url1=https://brokenriverbooks.com/feed/'

However if you change the port to 9876 it fails. For some reason the rssCloud plugin seems unable to hit ports that are not 80.

I've uncovered a couple of things regarding this.

1. The rsscloud plugin doesn't specify a scheme for the URLs so although it seems to work in some situations on wordpress.com on an up-to-date (6.1.1) self-hosted blog it always fails. "A valid URL was not provided"

2. wp_http_validate_url seems to only allow URLs with ports 80, 443, or 8080 so aggregators running on other random ports fail.

I logged into a test site with the RSS Cloud plugin (https://rsscloud.andysylvester.com/), then updated my test app to point to it (see https://gist.github.com/andysylvester/57400e6352529858a1dc8c1ced1f11b3), got the same response from the plugin that I got from my test WordPress.com site:

root@AndyDO-03:~/rssCloudSimple# node rssCloudTestOriginal06.js

testRssCloud v0.40 running on port 2222.

http://fedwiki.andysylvester.com

2222

/feedupdated

response: == ?xml version='1.0'?
notifyResult success='false' msg='Error testing notification URL : A valid URL was not provided.' (I removed the brackets around the XML response)
.

pleaseNotify: success == undefined, msg == "undefined"

Again, my test app was running on port 2222. I am guess that, from Andrew's earlier comment, port 80 probably works, but the plugin should be able to respond to other ports besides 80.

I'll see about reaching out to the plugin author about #1.

#2 will most likely be the norm but I'll mention it to the author as well.

Just saw your comment, could some other function besides wp_http_validate_url be used? Is it even necessary?

wp_http_validate is not used specifically by the plugin, but rather deep in the code, starting with the wp_safe_remote_get function. so they only want to make calls to "safe" URLs, so I'll mention it to Joseph Scott (who works at Automattic) in the email, but ultimately it will be his call.

Got it - thanks! Also, thanks for adding some comments on the WordPress.com forum post.

I updated my test app to use port 8080 per your earlier comment, the app was able to successfully register with the RSS Cloud plugin, and the test app got a response from the plugin when I made a post on the test WordPress site.

feedUpdated: url == https://rsscloud.andysylvester.com/feed/, now == 12/7/2022, 6:48:22 PM
theFeed == "rssCloud"
"Test post 048"

👋 Hi!

On the port front, I'm not inclined to change it. WordPress and the RSSCloud plugin have been working this way for more than 7 years.

The scheme issue seems to be new, or at least something that may have changed in the last 7 years. I'll unwind that code path more and see what falls out. The already suggested idea of defaulting to HTTP when no scheme is provided sounds like a reasonable way forward.

Separately, it looks like there are couple of spots in the plugin that need updating to run under PHP 8. I'll look into that closer and get an update out if needed.

Thank you!

@josephscott — nice to see you here… long time. 🤪

Joseph, thanks for participating in this thread! I think that the WordPress-supported ports that Andrew Shell discovered (80, 443, or 8080) should be added to the WordPress.com support page (https://wordpress.com/support/rsscloud/) and the RSS Cloud plugin page (https://wordpress.org/plugins/rsscloud/). This information is useful for aggregator developers (currently, FeedLand (http://feedland.org/) by Dave Winer is not recognizing RSS Cloud support from WordPress.com feeds, probably due to the port issue).

@andysylvester — can you say simply what the issue is from FeedLand’s point of view?

Here are items from my debug session (https://andysylvester.com/2022/12/04/rss-cloud-support-in-wordpress-com-not-working/):

===

2. Made a post (https://rsscloud4.wordpress.com/2022/12/04/test-post-007/) at 2:18 pm, no response to my test script.

3. Checked in FeedLand for new item, did not see it until 2:28pm (10 minutes later). In testing of other tools creating RSS feeds with rssCloud support, the item would appear in FeedLand within a few seconds. I have noticed that feed readers seem to take up to 30 minutes to detect that there has been a change in a RSS feed from WordPress.com or WordPress.org sites.

@andysylvester -- i'm no closer to understanding why it not work. if you have that info, please share. thanks.

@scripting -- I think that when FeedLand attempts to register a WordPress.com RSS feed with the associated site, or to register a WordPress.org RSS feed with a site using the RSS Cloud plugin, the registration fails, and so the RSS feed updates in FeedLand for these sites occurs when FeedLand detects that the RSS feed has changed (I assume through the use of polling).

In the debug post I included in my earlier comment and in this gist thread, I have demonstrated how registration can fail for the RSS Cloud server in WordPress.com and WordPress.org sites. Since I can update feeds using Drummer/Old School and FeedLand's feeds (which are both using Andrew Shell's rssCloud server in the cloud element in the RSS feed) and see the response in FeedLand within seconds, this leads me to think that (1) WordPress.com and WordPress.org sites can only accept registration from scripts/applications on ports 80, 443, or 8080, and that FeedLand is not sending registration on those ports, so the registration fails and updated posts do not show up within seconds of the post going live, or (2) there is some problem in the FeedLand rssCloud implementation. Since FeedLand is really the only feed reader application that supports rssCloud (aside from the recent work by Colin Walker (scripting/reallysimple#11)), and I can demonstrate that two other feed generating tools appear to work correctly with FeedLand rssCloud functionality, I don't think (2) is the correct answer. I think (1) is the correct answer.

@andysylvester -- what i'm taking away from this is that when FeedLand attempts to register with WordPress it fails.

Also FeedLand is not the only aggregator that uses rssCloud. Don't make assumptions that you have no data for.

Let's start here.

http://feeder.scripting.com/returnjson?feedurl=https://unberkeley.wordpress.com/feed/

feeder.scripting.com is a web service that shows you exactly what FeedLand will get when it reads the feed.

As you can see the cloud element is on port 80.

If I understand correctly, this feed will work because WordPress is working properly as long as it's port 80.

Correct?

https://rsscloud4.wordpress.com/feed/ and https://rsscloud.andysylvester.com/feed/

The issue is not with the content of the cloud element. I think the issue is that the WordPress rssCloud server will not accept the notification URL from an application if it is not on ports 80, 443, or 8080.

I think it is poor form to make a comment asking for help in a negative way and then delete it (as the email thread from this gist shows). Please stand behind your words. I would like to work this out, but when you do things like that, I get the feeling you want it all "your way". Since you are the developer of FeedLand, only you can see what is happening in the app beyond the Javascript console. I have supplied a lot of data showing my investigation of the problems with the WordPress rssCloud implementation, but I can only investigate things from "the outside" for the most part.

@scripting -- you are correct, River5 also supports rssCloud. If you know of other aggregators that support rssCloud, I would be interested in testing them against this WordPress issue.

Andy please I'm just trying to figure out what's going on. I deleted the question because I thought of a much better simpler way to ask it.

Steps I took:

1. Have a new blog hosted on WordPress.com (ex https://andrewshell.wordpress.com/)
2. Post a new blog post (ex https://andrewshell.wordpress.com/2022/12/09/test-post-1/)
3. Add RSS feed of blog to FeedLand (By pressing "+ Feed" button, pasting URL, then OK)
   1. Example feed https://andrewshell.wordpress.com/feed/
4. See the first blog post in FeedLand
5. Add a new blog post (ex https://andrewshell.wordpress.com/2022/12/09/test-post-2/)

What I expect to see:

1. Test post 2 should be listed in FeedLand shortly after posting it

What I saw:

1. Test post 2 took about an hour to show up in FeedLand

Andrew, that result is consistent with my debug session (https://andysylvester.com/2022/12/04/rss-cloud-support-in-wordpress-com-not-working/). I did some experimenting with River 5 as a debug tool this afternoon. When I ran it on ports 443 and 8080, I was able to see the RSS Cloud plugin site update within a few seconds (https://rsscloud.andysylvester.com/). However the WordPress.com site did not update (https://rsscloud4.wordpress.com/feed/). I will add some test code tomorrow to look at the WordPress.com rssCloud server response. My suspicion is that WordPress.com sites need to have the notification URL on port 80, but I will check this tomorrow.

Andrew -- I'm looking into your report now.

One tool you have to see how rssCloud is working re this feed is going to the Feed Info page for the feed.

http://feedland.org/?feedurl=https%3A%2F%2Fandrewshell.wordpress.com%2Ffeed%2F

The Last renew number is the last time FeedLand successfully renewed the notification request with the feed's cloud server.

At this time the number is 20 hours.

Just a piece of data.

You can also get FL to check the feed immediately, the conventional way by clicking on the Check Now link

A screen shot.

A report on my experiment in the next comment.

1. First, it's hard/impossible for me to step through this in the debugger because the local version of FeedLand that I use for testing is behind a firewall, and can't participate in rssCloud stuff. So I'm testing using the deployed server.

2. I did find and fix an error. It was getting a 301 response, permanent redirect After fixing, the redirect is taking place.

3. Another error shows up when renewing andrewshell.wordpress.com -- <notifyResult success='false' msg='No feed for url1.' />

going to look into this now

I added some debugging code, and had it display the request to the console before it sends it off to wordpress.com.

theRequest == {
    "url": "http://andrewshell.wordpress.com:80/?rsscloud=notify",
    "followAllRedirects": true,
    "maxRedirects": 5,
    "headers": {
        "Accept": "application/json"
    },
    "method": "POST",
    "form": {
        "port": "1670",
        "path": "/feedupdated",
        "url1": "https://andrewshell.wordpress.com/feed/",
        "protocol": "http-post"
    }
}

I think my next step is going to be to assume that it's the "form" that's tripping up the server? Couldn't hurt to use the default of body instead of form. This came up in recent work with the Mastodon API. Back in a bit...

BTW -- I see another problem. Every time feedland.js launches it might be assigned a different port.

This means that if it should crash and be relaunched, it might be assigned a different port when it is relaunched. Or when I install a new version as I am now doing every few minutes while I test this.

The upshot is that it will be sending notifications to the wrong port until the subscriptions are renewed within no more than 24 hours.

Another BTW -- this feature went in a few days before the product was feature frozen before release, and there was no one in the test group who had anything remotely like the experience Andrew has with rssCloud, so I did some simple testing and let it go, figuring at some point people who cared about rssCloud would show up and we'd do exactly what we're doing now. 😄

Dave, we determined from our tests against the WordPress plugin that it only works if the aggregator is on ports 80, 443, or 8080 otherwise, it considers it an unsafe URL. So sending port 1670 shouldn't work.

Here's the new request.

{
    "url": "http://andrewshell.wordpress.com:80/?rsscloud=notify",
    "method": "POST",
    "followAllRedirects": true,
    "maxRedirects": 5,
    "headers": {
        "Content-Type": "application/x-www-form-urlencoded"
    },
    "body": "port=1670&path=%2Ffeedupdated&url1=https%3A%2F%2Fandrewshell.wordpress.com%2Ffeed%2F&protocol=http-post"
}