Debugging rssCloud on WordPress.com

README.md

I have a test application running at test.rsscloud.io on ports 80 and 9876.

The following code works:

curl --location --request POST 'https://brokenriverbooks.com/?rsscloud=notify' \
--header 'Content-Type: application/x-www-form-urlencoded' \
--data-urlencode 'domain=test.rsscloud.io' \
--data-urlencode 'port=80' \
--data-urlencode 'path=/feedupdated-s8759' \
--data-urlencode 'registerProcedure=' \
--data-urlencode 'protocol=http-post' \
--data-urlencode 'url1=https://brokenriverbooks.com/feed/'

However if you change the port to 9876 it fails. For some reason the rssCloud plugin seems unable to hit ports that are not 80.

Let's start here.

http://feeder.scripting.com/returnjson?feedurl=https://unberkeley.wordpress.com/feed/

feeder.scripting.com is a web service that shows you exactly what FeedLand will get when it reads the feed.

As you can see the cloud element is on port 80.

If I understand correctly, this feed will work because WordPress is working properly as long as it's port 80.

Correct?

https://rsscloud4.wordpress.com/feed/ and https://rsscloud.andysylvester.com/feed/

The issue is not with the content of the cloud element. I think the issue is that the WordPress rssCloud server will not accept the notification URL from an application if it is not on ports 80, 443, or 8080.

I think it is poor form to make a comment asking for help in a negative way and then delete it (as the email thread from this gist shows). Please stand behind your words. I would like to work this out, but when you do things like that, I get the feeling you want it all "your way". Since you are the developer of FeedLand, only you can see what is happening in the app beyond the Javascript console. I have supplied a lot of data showing my investigation of the problems with the WordPress rssCloud implementation, but I can only investigate things from "the outside" for the most part.

@scripting -- you are correct, River5 also supports rssCloud. If you know of other aggregators that support rssCloud, I would be interested in testing them against this WordPress issue.

Andy please I'm just trying to figure out what's going on. I deleted the question because I thought of a much better simpler way to ask it.

Steps I took:

1. Have a new blog hosted on WordPress.com (ex https://andrewshell.wordpress.com/)
2. Post a new blog post (ex https://andrewshell.wordpress.com/2022/12/09/test-post-1/)
3. Add RSS feed of blog to FeedLand (By pressing "+ Feed" button, pasting URL, then OK)
   1. Example feed https://andrewshell.wordpress.com/feed/
4. See the first blog post in FeedLand
5. Add a new blog post (ex https://andrewshell.wordpress.com/2022/12/09/test-post-2/)

What I expect to see:

1. Test post 2 should be listed in FeedLand shortly after posting it

What I saw:

1. Test post 2 took about an hour to show up in FeedLand

Andrew, that result is consistent with my debug session (https://andysylvester.com/2022/12/04/rss-cloud-support-in-wordpress-com-not-working/). I did some experimenting with River 5 as a debug tool this afternoon. When I ran it on ports 443 and 8080, I was able to see the RSS Cloud plugin site update within a few seconds (https://rsscloud.andysylvester.com/). However the WordPress.com site did not update (https://rsscloud4.wordpress.com/feed/). I will add some test code tomorrow to look at the WordPress.com rssCloud server response. My suspicion is that WordPress.com sites need to have the notification URL on port 80, but I will check this tomorrow.

Andrew -- I'm looking into your report now.

One tool you have to see how rssCloud is working re this feed is going to the Feed Info page for the feed.

http://feedland.org/?feedurl=https%3A%2F%2Fandrewshell.wordpress.com%2Ffeed%2F

The Last renew number is the last time FeedLand successfully renewed the notification request with the feed's cloud server.

At this time the number is 20 hours.

Just a piece of data.

You can also get FL to check the feed immediately, the conventional way by clicking on the Check Now link

A screen shot.

A report on my experiment in the next comment.

1. First, it's hard/impossible for me to step through this in the debugger because the local version of FeedLand that I use for testing is behind a firewall, and can't participate in rssCloud stuff. So I'm testing using the deployed server.

2. I did find and fix an error. It was getting a 301 response, permanent redirect After fixing, the redirect is taking place.

3. Another error shows up when renewing andrewshell.wordpress.com -- <notifyResult success='false' msg='No feed for url1.' />

going to look into this now

I added some debugging code, and had it display the request to the console before it sends it off to wordpress.com.

theRequest == {
    "url": "http://andrewshell.wordpress.com:80/?rsscloud=notify",
    "followAllRedirects": true,
    "maxRedirects": 5,
    "headers": {
        "Accept": "application/json"
    },
    "method": "POST",
    "form": {
        "port": "1670",
        "path": "/feedupdated",
        "url1": "https://andrewshell.wordpress.com/feed/",
        "protocol": "http-post"
    }
}

I think my next step is going to be to assume that it's the "form" that's tripping up the server? Couldn't hurt to use the default of body instead of form. This came up in recent work with the Mastodon API. Back in a bit...

BTW -- I see another problem. Every time feedland.js launches it might be assigned a different port.

This means that if it should crash and be relaunched, it might be assigned a different port when it is relaunched. Or when I install a new version as I am now doing every few minutes while I test this.

The upshot is that it will be sending notifications to the wrong port until the subscriptions are renewed within no more than 24 hours.

Another BTW -- this feature went in a few days before the product was feature frozen before release, and there was no one in the test group who had anything remotely like the experience Andrew has with rssCloud, so I did some simple testing and let it go, figuring at some point people who cared about rssCloud would show up and we'd do exactly what we're doing now. 😄

Dave, we determined from our tests against the WordPress plugin that it only works if the aggregator is on ports 80, 443, or 8080 otherwise, it considers it an unsafe URL. So sending port 1670 shouldn't work.

Here's the new request.

{
    "url": "http://andrewshell.wordpress.com:80/?rsscloud=notify",
    "method": "POST",
    "followAllRedirects": true,
    "maxRedirects": 5,
    "headers": {
        "Content-Type": "application/x-www-form-urlencoded"
    },
    "body": "port=1670&path=%2Ffeedupdated&url1=https%3A%2F%2Fandrewshell.wordpress.com%2Ffeed%2F&protocol=http-post"
}

And the wordpress server is responding, as before, with <notifyResult success='false' msg='No feed for url1.'>

At this point I don't see anything more I can do, it looks to me like the call is correct, and the error message seems to say it didn't find the URL in the request.

I'll look to see if I can identify anything wrong with the request call. The one thing that I know won't work is using port 1670 because of the unsafe URL filter that had been discussed previously.

@andrewshell -- aha. okay that explains what's going on.

@josephscott and I worked out this protocol quite a few years ago and he may still be tuned in.

It's easy to imagine another piece of software putting different constraints on what port FeedLand can run on.

The odd thing is that feedland.org is as far as anyone outside the server is concerned running on port 80. I wonder what would happen if I just forced the port to be 80. Do you think they're calling me back by IP address or by a reverse DNS call or..?

I love this bullshit, it's fun (no sarcasm). 😄

Or force it to be 80 if the domain we're talking to is on wordpress.com! ;-)

Oh the humanity.

Also if that is the problem they surely could provide a better error message. I thought from the message they returned that it was a syntax error of some kind.

Doing port 80 might work. I did notice that feedland doesn't seem to support the GET verification path (with challenge param) for pleaseNotify with the domain specified. (See http://walkthrough.rsscloud.co/#challengeParameter)

@andrewshell -- okay that goes on my todo list. :-)

i do have another project that i'm working on that i must move forward today, and i want to think about my next steps here, because i don't want to hack something in here, it has to be maintainable and documented, otherwise it's sure to break in a few months when i forgot what i did here. feedland is a very diverse piece of software and docs and maintainability are huge priorities.

andrew thanks for your help with this. i really enjoy working with you.

@scripting No rush on this, I agree it needs to be done correctly. One thing I found with testing is that the error you're getting might be an issue with request, and might even be deliberate. I haven't dug into its source code yet, but it appears that even when followAllRedirects is true, it doesn't look like the body of the POST is present when the second call is made.

If you make the same call against https://andrewshell.wordpress.com/?rsscloud=notify instead (notice the https) it returns <notifyResult success='false' msg='Error testing notification URL : A valid URL was not provided.' /> which was because of the port number. I think having the form parameter is cleaner and wasn't the culprit.

The unfortunate solution is probably not to use followAllRedirects: true and watch for 301 and 302 status codes, and resend yourself with the location value in the header.

By using this wrapper instead of calling request directly it looks like the call does what you'd expect.

function requestFollowRedirects(theRequest, callback) {
    theRequest.followAllRedirects = false;
    request (theRequest, function (err, response, body) {
        if (parseInt(theRequest.maxRedirects) > 0 && [301, 320].includes(response.statusCode) && response.headers.location != null) {
            const newRequest = Object.assign({}, theRequest, { url: response.headers.location });
            newRequest.maxRedirects--;
            requestFollowRedirects (newRequest, callback);
            }
        callback (err, response, body)
        });
    }

Andrew, thanks for sharing your work! I appreciate it. I am still planning to do some testing today, but it is apparent to me from your investigation that processing of rssCloud feeds from WordPress.com and WordPress.org sites is going to require some additional work beyond what is currently in FeedLand, and probably also in River5.

@andrewshell -- I've made the change in how FeedLand requests notification, it does its own redirection as illustrated by your example.

I am about to deploy it on feedland.org.

but i don't have enough information about the wordpress server to know what domain name it's using to address my server, for all i know they're using the ip address. if they use anything other than feedland.org to send the notification, port 80 will not work.

i might be willing to do a little customization just for wordpress.com servers, but not unless i know what i'm doing. i'm not willing to even experiment with this on a live server. so we're kind of at an impasse. but at least it should work with your rsscloud server, which i assume does not care what port i'm running on, so there's that. ;-)

The WordPress server follows the rssCloud spec. So since you don't specify the domain in your call, it does a POST to your IP address with the specified port. If you want it to go to feedland.org:80, you'll need to specify the domain name and handle the appropriate validate endpoint where it's a GET request with url and challenge parameters. See http://walkthrough.rsscloud.co/#challengeParameter

@andrewshell -- thank you. that's the info i was looking for. it's as if i didn't write those words myself. 😄

For reference, the WordPress safe request calls being limited to 80, 443, and 8080 happen inside WordPress core at https://core.trac.wordpress.org/browser/trunk/src/wp-includes/http.php#L589 - so this applies broadly across WordPress installs, including WordPress.com.

I have a new version of feedland.org running, it makes a pleaseNotify request using this request object.

{
    "url": "http://andrewshell.wordpress.com:80/?rsscloud=notify",
    "method": "POST",
    "followAllRedirects": true,
    "maxRedirects": 5,
    "headers": {
        "Content-Type": "application/x-www-form-urlencoded"
    },
    "body": "domain=feedland.org&port=80&path=%2Ffeedupdated&url1=https%3A%2F%2Fandrewshell.wordpress.com%2Ffeed%2F&protocol=http-post"
}

The upshot of this, as far as I know -- we should now be cool with wordpress.com's servers.

The way to test it, I guess is with a wordpress.com hosted site, try renewing its subscription and see what happens?

Is this correct?