Debugging rssCloud on WordPress.com

README.md

I have a test application running at test.rsscloud.io on ports 80 and 9876.

The following code works:

curl --location --request POST 'https://brokenriverbooks.com/?rsscloud=notify' \
--header 'Content-Type: application/x-www-form-urlencoded' \
--data-urlencode 'domain=test.rsscloud.io' \
--data-urlencode 'port=80' \
--data-urlencode 'path=/feedupdated-s8759' \
--data-urlencode 'registerProcedure=' \
--data-urlencode 'protocol=http-post' \
--data-urlencode 'url1=https://brokenriverbooks.com/feed/'

However if you change the port to 9876 it fails. For some reason the rssCloud plugin seems unable to hit ports that are not 80.

Also if that is the problem they surely could provide a better error message. I thought from the message they returned that it was a syntax error of some kind.

Doing port 80 might work. I did notice that feedland doesn't seem to support the GET verification path (with challenge param) for pleaseNotify with the domain specified. (See http://walkthrough.rsscloud.co/#challengeParameter)

@andrewshell -- okay that goes on my todo list. :-)

i do have another project that i'm working on that i must move forward today, and i want to think about my next steps here, because i don't want to hack something in here, it has to be maintainable and documented, otherwise it's sure to break in a few months when i forgot what i did here. feedland is a very diverse piece of software and docs and maintainability are huge priorities.

andrew thanks for your help with this. i really enjoy working with you.

@scripting No rush on this, I agree it needs to be done correctly. One thing I found with testing is that the error you're getting might be an issue with request, and might even be deliberate. I haven't dug into its source code yet, but it appears that even when followAllRedirects is true, it doesn't look like the body of the POST is present when the second call is made.

If you make the same call against https://andrewshell.wordpress.com/?rsscloud=notify instead (notice the https) it returns <notifyResult success='false' msg='Error testing notification URL : A valid URL was not provided.' /> which was because of the port number. I think having the form parameter is cleaner and wasn't the culprit.

The unfortunate solution is probably not to use followAllRedirects: true and watch for 301 and 302 status codes, and resend yourself with the location value in the header.

By using this wrapper instead of calling request directly it looks like the call does what you'd expect.

function requestFollowRedirects(theRequest, callback) {
    theRequest.followAllRedirects = false;
    request (theRequest, function (err, response, body) {
        if (parseInt(theRequest.maxRedirects) > 0 && [301, 320].includes(response.statusCode) && response.headers.location != null) {
            const newRequest = Object.assign({}, theRequest, { url: response.headers.location });
            newRequest.maxRedirects--;
            requestFollowRedirects (newRequest, callback);
            }
        callback (err, response, body)
        });
    }

Andrew, thanks for sharing your work! I appreciate it. I am still planning to do some testing today, but it is apparent to me from your investigation that processing of rssCloud feeds from WordPress.com and WordPress.org sites is going to require some additional work beyond what is currently in FeedLand, and probably also in River5.

@andrewshell -- I've made the change in how FeedLand requests notification, it does its own redirection as illustrated by your example.

I am about to deploy it on feedland.org.

but i don't have enough information about the wordpress server to know what domain name it's using to address my server, for all i know they're using the ip address. if they use anything other than feedland.org to send the notification, port 80 will not work.

i might be willing to do a little customization just for wordpress.com servers, but not unless i know what i'm doing. i'm not willing to even experiment with this on a live server. so we're kind of at an impasse. but at least it should work with your rsscloud server, which i assume does not care what port i'm running on, so there's that. ;-)

The WordPress server follows the rssCloud spec. So since you don't specify the domain in your call, it does a POST to your IP address with the specified port. If you want it to go to feedland.org:80, you'll need to specify the domain name and handle the appropriate validate endpoint where it's a GET request with url and challenge parameters. See http://walkthrough.rsscloud.co/#challengeParameter

@andrewshell -- thank you. that's the info i was looking for. it's as if i didn't write those words myself. 😄

For reference, the WordPress safe request calls being limited to 80, 443, and 8080 happen inside WordPress core at https://core.trac.wordpress.org/browser/trunk/src/wp-includes/http.php#L589 - so this applies broadly across WordPress installs, including WordPress.com.

I have a new version of feedland.org running, it makes a pleaseNotify request using this request object.

{
    "url": "http://andrewshell.wordpress.com:80/?rsscloud=notify",
    "method": "POST",
    "followAllRedirects": true,
    "maxRedirects": 5,
    "headers": {
        "Content-Type": "application/x-www-form-urlencoded"
    },
    "body": "domain=feedland.org&port=80&path=%2Ffeedupdated&url1=https%3A%2F%2Fandrewshell.wordpress.com%2Ffeed%2F&protocol=http-post"
}

The upshot of this, as far as I know -- we should now be cool with wordpress.com's servers.

The way to test it, I guess is with a wordpress.com hosted site, try renewing its subscription and see what happens?

Is this correct?

So I subscribed to this feed.

http://feedland.org/?feedurl=https%3A%2F%2Funberkeley.wordpress.com%2Ffeed%2F

Now I'm going to add a test post and see what happens.

Note I couldn't follow this in the debugger so I don't know if we correctly handled the challenge parameter.

Houston we have liftoff.

I was able to test with my WordPress blog and can concur that it appears to be working now.

Andrew, I just did a test with my WordPress.com blog (https://rsscloud4.wordpress.com/feed/), and it took 60 minutes for the post to appear on FeedLand. Could you perform another test with your feed (https://andrewshell.wordpress.com/feed/)? I will do some additional tests with my WordPress.com blog and my WordPress.org blog with the RSS Cloud plugin. Based on my test, it seems that FeedLand is still having problems with RSS Cloud for WordPress.com blogs.

@andysylvester I tried it again, and it's still working. It's possible you ran your test before FeedLand resubscribed. I'd test it again if I were you.

BTW you can see how long it's been since the last cloud renew by going to the Feed Info page for the feed.

http://feedland.org/?feedurl=https%3A%2F%2Frsscloud4.wordpress.com%2Ffeed%2F

In this case it was successfully renewed 13 hours ago.

It sounds like things are working now, at least with WordPress.com feeds.

The RSS Cloud plugin itself does still need to be updated. I'll get an update for that out this week.

I've released an updated version of the RSSCloud plugin ( 0.5.0 ) - https://wordpress.org/plugins/rsscloud/ - that includes the PHP 8+ fixes and the default scheme of HTTP when none is provided. Let me know if anything else comes up.

@scripting, thanks for the pointer on the cloud renew time, that is helpful.

I did two test posts from my WordPress.com site just now (https://rsscloud4.wordpress.com/feed/), both posts appeared in FeedLand within several seconds, so FeedLand support of rssCloud is working for me now. Thanks @scripting and @andrewshell for your help!

@josephscott - I will go ahead and download the new plugin and give it a test - thanks!

@josephscott - I updated to version 0.5 on my plugin test site (https://rsscloud.andysylvester.com/), made two posts, they have not appeared on FeedLand yet. Is there anything you would like me to check? The FeedLand sub to this site was renewed 2 hrs ago, should I wait for the next renewal to try again?

First thing I'd check is to review the error logs for the site, just to make sure nothing went sideways.

Next, is there anything on the site that would interfere with the WordPress cron feature? And does the site get plenty of page views, to give an opportunity for the cron feature to run.

It might also be worthwhile checking on the cron jobs inside the WordPress install, a plugin like https://wordpress.org/plugins/wp-crontrol/ should be sufficient for that.

@josephscott I will look at those items, thanks for the suggestions!

Good morning! I decided that now that I have a working implementation again and it's fresh in my mind, I should write an example app in JavaScript that implements the aggregator side of rssCloud.

I believe I have it working for notifications via POST, but have hit a snag with the notifications via GET.

My code assumes there will be a challenge parameter, and that it will not be used for a "real" notification of an update, that it will only be used as part of the notification request. But it seems it is being used for notification, and when that happens there is no challenge parameter (which would make sense).

The example I'm testing with is unberkeley.wordpress.com.

I'm going to assume that the description above is what's going on, and be prepared for a call without the challenge parameter.

If you want to see the actual code it's in this file.

https://github.com/scripting/reallysimple/blob/main/demos/clouddemo/clouddemo.js

BTW, in reviewing the walkthrough -- I desperately want to rewrite it. It has become very dated, and the links to examples are broken. However I'm not sure I will have the time for that, all the more reason it's important to have a good working example out there.

Update: As I'm debugging it further -- the challenge parameter does appear to be there even for real pings, not test pings. Still looking into it.

It only took me an hour to find a missing break; statement.

I've been programming for 800 years and I still make beginner's mistakes.

Never mind. ;-)

I couldn't leave a comment on the reallySimple repo for clouddemo.js (it appears to be locked), so I am leaving a comment here with my report. I was able to run the app on port 443 and successfully register a WordPress.com site and a WordPress.org site using version 0.5.0 of the WordPress RSS Cloud plugin. When I ran my test script on the WordPress.org site, it worked, but I figured out that the WordPress.org site is caching the RSS feed, which explains why feed updates were not seen by FeedLand.

@josephscott - I have looked at this Stack Exchange post for ideas on disabling caching of WordPress feeds, if you have any better ideas, I am interested.

The code for clouddemo.js looks good, I am pleased that it could run on port 443, since I already had an app running on port 80 on my server and did not want to change it.

@andysylvester are your referring to the RSS blog/widget caching the results of a feed? That would be the other end of this, you would need something to trigger an invalidation of the cached feed.