Skip to content

Instantly share code, notes, and snippets.

@zachlim98

zachlim98/blog.md Secret

Created Nov 18, 2020
Embed
What would you like to do?
title categories
How to get the IBD50 list for FREE
data
trading

Introduction

The Investors' Business Daily is a newspaper covering the stock market, set up by William O'Neil in 1984. One of the most famous things that come from IBD is the IBD 50 - a list of the top 50 stocks that have been screened by the CAN SLIM method. The CAN SLIM method is a stock fundamental and technical screening methodology that was developed by William O'Niel.

So, one can do the screening manually using the methods described OR one can subscribe to the IBD. As part of one's IBD subscription, one would receive the weekly updates of the IBD 50. But, what if one just wants to know the IBD 50 without having the rest of the perks that come with an IBD subscription? Well, in 2015, innovator ETFs released an ETF (ticker: FFTY) that tracked the IBD50! And since the holdings of the ETF are available publicly, we can just scrap the holdings of FFTY in order to know each week what holdings are recommended by the CAN SLIM method

Disclaimer: An IBD subscription, I'm sure, comes with a lot more than the IBD50. This is merely a subset of what the subscription offers and so please feel free to subscribe to the full IBD if you find it helpful! For me, I felt I just needed the IBD50 list

Scrapping Process

Problems with BeautifulSoup

image

So, we all know that BeautifulSoup is the go-to when scrapping the web. However, I needed to scrap from zacks.com and the issue with that is that zacks.com stores the "holdings" in a JavaScript table. The second issue is that the initial page only displays 10 entries (out of the 50 stocks that are recommended under the IBD50 list), You can't really manipulate the webpage address to show the rest because, again, the holdings are stored in some JavaScript table. It was time to turn to the other favorite tool - selenium

Loading the webpage with Selenium

https://gist.github.com/48f5785bcfeeefc46687ea27a3255a9b

Because I was lazy and did not want to add the geckodriver to PATH (which you can find how to do here), I needed to specify my executable_path. (ps: the R in front is for Python to read it as a raw string and hence ignore the slash)

Once that was done, I simply set the URL for the FFTY holdings and used the driver to open the webpage.

https://gist.github.com/8a5b23bd1a605a9ea671d4fae1e2a0d7

Accessing a Dropdown box

Once that was done, I needed to access the dropdown list and change it such that it would show all 50 entries. The annoying about this, was that I kept getting an error that the element (the dropdown box) was blocked and not selectable! It turns out that the "Accept Cookies" form covered the dropdown box when the page initially loaded.

image

This was easily solved by using window.scroll to scroll it down about 400p (out of 1080p) so that the cookies form was out of the way. With that done, I simply had to find the dropdown box element and select '50'

https://gist.github.com/fc0c55daaa60a4bb14cd795636c4b265

Storing the data

This was the easiest part. After displaying all 50 symbols, I just had to use their CSS selector and pull the ticker symbol as well as their weights in the portfolio. I then used Yahoo Finance to pull their prices (just to make my life slightly easier).

https://gist.github.com/af68eac7acde77f793610f7df3fcb222

https://gist.github.com/1e365a105b3576eafc281138a16c2b1e

image

Conclusion

So there you have it! Super easy, super fast way to get the IBD50 list without having to pay for a full subscription if you ever so desire! Thanks for reading!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment