Code to retrieve all links from a given URL

get_all_links.py

import requests
from bs4 import BeautifulSoup

def get_all_links(url):
    content = requests.get(url).content
    soup = BeautifulSoup(content, 'html.parser')
    links = []
    for link in soup.find_all('a'):
        if link.has_attr('href'):
            href = link['href']
            if href and not href.startswith('#'):
                links.append(href)
    return links

if __name__ == '__main__':
    url = 'https://www.ashish.ch'
    links = get_all_links(url)
    print(links)

thanks for this! btw line 6 should probably be:

for link in BeautifulSoup(content, parse_only=SoupStrainer('a'), features='lxml'):

Details

as there is a warning:

GuessedAtParserWarning: No parser was explicitly specified, so I'm using the best available HTML parser for this system ("lxml"). This usually isn't a problem, but if you run this code on another system, or in a different virtual environment, it may use a different parser and behave differently.

The code that caused this warning is on line 24 of the file getTheories.py. To get rid of this warning, pass the additional argument 'features="lxml"' to the BeautifulSoup constructor.

(This is the list of available parsers https://www.crummy.com/software/BeautifulSoup/bs4/doc/#specifying-the-parser-to-use)

hi, if I wanted to make a list of all of these hrefs- how would I go about that?

make an empty list (hrefs = []) on line 5, and change line 8 to hrefs.append(link['href'])

Hi I am getting this error
Traceback (most recent call last):
File "c:\Users\yyy\Downloads\assignment\testwebscrapping.py", line 69, in
print(link['href'])
File "C:\Users\yyy\AppData\Roaming\Python\Python39\site-packages\bs4\element.py", line 1519, in getitem
return self.attrs[key]
KeyError: 'href'

updated the code to handle a few edge cases