Debugging rssCloud on WordPress.com

README.md

I have a test application running at test.rsscloud.io on ports 80 and 9876.

The following code works:

curl --location --request POST 'https://brokenriverbooks.com/?rsscloud=notify' \
--header 'Content-Type: application/x-www-form-urlencoded' \
--data-urlencode 'domain=test.rsscloud.io' \
--data-urlencode 'port=80' \
--data-urlencode 'path=/feedupdated-s8759' \
--data-urlencode 'registerProcedure=' \
--data-urlencode 'protocol=http-post' \
--data-urlencode 'url1=https://brokenriverbooks.com/feed/'

However if you change the port to 9876 it fails. For some reason the rssCloud plugin seems unable to hit ports that are not 80.

For reference, the WordPress safe request calls being limited to 80, 443, and 8080 happen inside WordPress core at https://core.trac.wordpress.org/browser/trunk/src/wp-includes/http.php#L589 - so this applies broadly across WordPress installs, including WordPress.com.

I have a new version of feedland.org running, it makes a pleaseNotify request using this request object.

{
    "url": "http://andrewshell.wordpress.com:80/?rsscloud=notify",
    "method": "POST",
    "followAllRedirects": true,
    "maxRedirects": 5,
    "headers": {
        "Content-Type": "application/x-www-form-urlencoded"
    },
    "body": "domain=feedland.org&port=80&path=%2Ffeedupdated&url1=https%3A%2F%2Fandrewshell.wordpress.com%2Ffeed%2F&protocol=http-post"
}

The upshot of this, as far as I know -- we should now be cool with wordpress.com's servers.

The way to test it, I guess is with a wordpress.com hosted site, try renewing its subscription and see what happens?

Is this correct?

So I subscribed to this feed.

http://feedland.org/?feedurl=https%3A%2F%2Funberkeley.wordpress.com%2Ffeed%2F

Now I'm going to add a test post and see what happens.

Note I couldn't follow this in the debugger so I don't know if we correctly handled the challenge parameter.

Houston we have liftoff.

I was able to test with my WordPress blog and can concur that it appears to be working now.

Andrew, I just did a test with my WordPress.com blog (https://rsscloud4.wordpress.com/feed/), and it took 60 minutes for the post to appear on FeedLand. Could you perform another test with your feed (https://andrewshell.wordpress.com/feed/)? I will do some additional tests with my WordPress.com blog and my WordPress.org blog with the RSS Cloud plugin. Based on my test, it seems that FeedLand is still having problems with RSS Cloud for WordPress.com blogs.

@andysylvester I tried it again, and it's still working. It's possible you ran your test before FeedLand resubscribed. I'd test it again if I were you.

BTW you can see how long it's been since the last cloud renew by going to the Feed Info page for the feed.

http://feedland.org/?feedurl=https%3A%2F%2Frsscloud4.wordpress.com%2Ffeed%2F

In this case it was successfully renewed 13 hours ago.

It sounds like things are working now, at least with WordPress.com feeds.

The RSS Cloud plugin itself does still need to be updated. I'll get an update for that out this week.

I've released an updated version of the RSSCloud plugin ( 0.5.0 ) - https://wordpress.org/plugins/rsscloud/ - that includes the PHP 8+ fixes and the default scheme of HTTP when none is provided. Let me know if anything else comes up.

@scripting, thanks for the pointer on the cloud renew time, that is helpful.

I did two test posts from my WordPress.com site just now (https://rsscloud4.wordpress.com/feed/), both posts appeared in FeedLand within several seconds, so FeedLand support of rssCloud is working for me now. Thanks @scripting and @andrewshell for your help!

@josephscott - I will go ahead and download the new plugin and give it a test - thanks!

@josephscott - I updated to version 0.5 on my plugin test site (https://rsscloud.andysylvester.com/), made two posts, they have not appeared on FeedLand yet. Is there anything you would like me to check? The FeedLand sub to this site was renewed 2 hrs ago, should I wait for the next renewal to try again?

First thing I'd check is to review the error logs for the site, just to make sure nothing went sideways.

Next, is there anything on the site that would interfere with the WordPress cron feature? And does the site get plenty of page views, to give an opportunity for the cron feature to run.

It might also be worthwhile checking on the cron jobs inside the WordPress install, a plugin like https://wordpress.org/plugins/wp-crontrol/ should be sufficient for that.

@josephscott I will look at those items, thanks for the suggestions!

Good morning! I decided that now that I have a working implementation again and it's fresh in my mind, I should write an example app in JavaScript that implements the aggregator side of rssCloud.

I believe I have it working for notifications via POST, but have hit a snag with the notifications via GET.

My code assumes there will be a challenge parameter, and that it will not be used for a "real" notification of an update, that it will only be used as part of the notification request. But it seems it is being used for notification, and when that happens there is no challenge parameter (which would make sense).

The example I'm testing with is unberkeley.wordpress.com.

I'm going to assume that the description above is what's going on, and be prepared for a call without the challenge parameter.

If you want to see the actual code it's in this file.

https://github.com/scripting/reallysimple/blob/main/demos/clouddemo/clouddemo.js

BTW, in reviewing the walkthrough -- I desperately want to rewrite it. It has become very dated, and the links to examples are broken. However I'm not sure I will have the time for that, all the more reason it's important to have a good working example out there.

Update: As I'm debugging it further -- the challenge parameter does appear to be there even for real pings, not test pings. Still looking into it.

It only took me an hour to find a missing break; statement.

I've been programming for 800 years and I still make beginner's mistakes.

Never mind. ;-)

I couldn't leave a comment on the reallySimple repo for clouddemo.js (it appears to be locked), so I am leaving a comment here with my report. I was able to run the app on port 443 and successfully register a WordPress.com site and a WordPress.org site using version 0.5.0 of the WordPress RSS Cloud plugin. When I ran my test script on the WordPress.org site, it worked, but I figured out that the WordPress.org site is caching the RSS feed, which explains why feed updates were not seen by FeedLand.

@josephscott - I have looked at this Stack Exchange post for ideas on disabling caching of WordPress feeds, if you have any better ideas, I am interested.

The code for clouddemo.js looks good, I am pleased that it could run on port 443, since I already had an app running on port 80 on my server and did not want to change it.

@andysylvester are your referring to the RSS blog/widget caching the results of a feed? That would be the other end of this, you would need something to trigger an invalidation of the cached feed.

@josephscott I am not sure what you mean by "RSS blog/widget". My understanding is that WordPress uses SimplePie to generate the RSS feed of a WordPress website/blog. It appears to be common knowledge that WordPress itself caches the RSS feed of the site (see https://amandagiles.com/blog/code-snippets/updating-wordpress-rss-feed-cache-time/). To me, this is a problem for using rssCloud with WordPress, since most feed readers will not display new updates from the site if the feed it retrieves has not changed, even if it receives a response from an rssCloud server that the site has updated. In my opinion, for rssCloud use, the caching should be turned off or reduced to 5-10 seconds.

Here is my anecdotal evidence supporting this: I made posts on my WordPress.org site using version 0.5.0 of the RSS Cloud plugin. I made 3 posts and received three responses from the plugin (see this gist). However, the posts did not appear in FeedLand. I checked the URL of the feed in my browser (https://rsscloud.andysylvester.com/feed/), it was updated. I then did a curl command of the feed, and it had not updated. I then went to the Dashboard for my site, went to Settings/Readings, and changed the value of "Syndication feeds show the most recent x items" from 10 to 5. After doing that, the curl command returned the feed I was seeing in my browser, and eventually the posts showed up in FeedLand.

Following up on your last sentence, I think that the RSS Cloud plugin should trigger an invalidation of the cached feed so that feed readers will get the most recent version of the feed. Otherwise, the RSS Cloud plugin notification does not provide any benefit.

Sorry, my previous comment was supposed to be "RSS block/widget".

In terms of the RSS feed updating, I've run a few tests on a stock out of the box WordPress site and didn't see any problems with the RSS feed updating right away.

What you described with the feed being updated when you viewed it with the browser but not updated when done with curl sounds like it could be a caching difference based on cookies. Were you logged into the site with the browser you used to check the feed?

I went back to the fresh stock WordPress install for testing and still didn't have a problem with the feed updating when viewed logged out or logged in, so I don't think this is sufficient to explain what you saw either.

Are there any other plugins on the site you are using to test? Can you try it with a fresh WordPress install with no plugins and check to see if the RSS feed updates as expected.

@josephscott @andysylvester --

Once I got the FeedLand implementation working with WordPress and @andrewshell's server, I created a very simple demo app that just does the rssCloud functionality in FeedLand anyone can test with our implementation, and see exactly what's going on. MIT License.

https://github.com/scripting/reallysimple/tree/main/demos/clouddemo

It also provides a reference implementation for people just getting started.

At some point I need to rewrite the walkthrough. It's not very well done imho (I am the author).

@josephscott - I was logged into the site, so I will check on that. I really appreciate your feedback, and will work on testing today.

@scripting - I was already aware of your implementation (see this link to comment above), so I assume you are really leaving your comment for the benefit of @josephscott. I used your app to help figure out my current issue with the RSS Cloud plugin. Since it appears you have removed your block of my account on the reallySimple repo, I will post further feedback on your reference app on that repo in the future.

Well, I see it didn't take long to restore the block of my account...

@josephscott - Yesterday, per your suggestion, I created a new WordPress site (https://scott2.andysylvester.com/) and disabled all plugins, then installed version 0.5.0 of the RSS Cloud plugin, and was able to successfully see posts appear within several seconds on FeedLand, which would indicate that the plugin is working. I also checked the feed in the browser and using curl, and both contained the correct content. This morning, I decided to do additional tests using my test app and Dave Winer's test app. Both apps were able to receive the notification from new posts. However, posts did not appear in FeedLand. I also checked the feed in the browser and using curl, and both contained the correct content.

When I looked at the output of my app, which uses the readFeed function within the NPM reallySimple package to get the feed, the first title in the feed it read was for test post 8, although I had created a new test post. This behavior continued even though I made several additional posts. In the posts, I captured info on each state of testing. To me, if FeedLand was not seeing any changes in the feed, I can totally understand that it would not display anything from the feed. As I have commented before, this seems like a caching problem, but I am not sure where to look.

I do not fully understand what is happening here, but this behavior has now occurred on two different WordPress sites using the latest RSS Cloud plugin. I would like to work this out, because WordPress is the major creator of rssCloud-enabled feeds, and I want to use rssCloud and WordPress in a new project. @josephscott, I am willing to keep posting to this thread, or in some other forum to work this out, I appreciate the feedback you have given me, and want to run this problem down if I can.

From what you are describing, I'm not sure that this is directly related to the RSS Cloud plugin. You mentioned that it worked out of the box, with the feed showing the latest data and pings going out to FeedLand.

To track this down I think you are going to need more data. I'd recommend adding logging to every step, that what you can confirm when a step went missing that you were expecting to happen.

@josephscott - thanks for getting back with me. I did see it work "out of the box", but then saw this caching issue when I was running some supplemental test scripts. I will add some logging and report back on what I find.