Last active
May 11, 2024 03:17
-
-
Save applch/2f55c39510b37338b26eb046cab672d3 to your computer and use it in GitHub Desktop.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| About Applebot | |
| Applebot is the web crawler for Apple. Products like Siri and Spotlight Suggestions use Applebot. | |
| Identifying Applebot | |
| Traffic coming from Applebot is identified by its user agent, and reverse DNS shows it in the *.applebot.apple.com domain, originating from the 17.0.0.0 net block. | |
| Verifying that traffic is from Applebot | |
| In macOS, the host command can be used to determine if an IP address is part of Applebot. These examples show the host command and its result: | |
| $ host 17.58.101.179 | |
| 179.101.58.17.in-addr.arpa domain name pointer 17-58-101-179.applebot.apple.com. | |
| The host command can also be used to verify that the DNS points to the same IP address: | |
| $ host 17-58-101-179.applebot.apple.com | |
| 17-58-101-179.applebot.apple.com has address 17.58.101.179. | |
| Verifying Applebot user agent | |
| The user-agent string contains ”Applebot” and other information. This is the format: | |
| Mozilla/5.0 (Device; OS_version) AppleWebKit/WebKit_version (KHTML, like Gecko) | |
| Version/Safari_version Safari/WebKit_version (Applebot/Applebot_version) | |
| Examples for desktop: | |
| Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 | |
| (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1) | |
| Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 | |
| (KHTML, like Gecko) Version/12.1.1 Safari/605.1.15 (Applebot/0.1) | |
| Examples for mobile: | |
| Mozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 | |
| (KHTML, like Gecko) Version/8.0 Mobile/12B410 Safari/600.1.4 (Applebot/0.1; | |
| +http://www.apple.com/go/applebot) | |
| Mozilla/5.0 (iPhone; CPU iPhone OS 13_4_1 like Mac OS X) AppleWebKit/605.1.15Z | |
| (KHTML, like Gecko) Version/13.1 Mobile/15E148 Safari/604.1 (Applebot/0.1) | |
| Customizing robot.txt rules | |
| Applebot respects standard robots.txt directives that are targeted at Applebot. In this example, Applebot doesn't try to crawl documents that are under /private/ or /not-allowed/: | |
| User-agent: Applebot # apple | |
| Allow: / # Allow everything (also true if omitted) | |
| Disallow: /private/ # disallow this directory | |
| User-agent: * # any robot | |
| Disallow: /not-allowed/ # disallow this directory | |
| If robots instructions don't mention Applebot but do mention Googlebot, the Apple robot will follow Googlebot instructions. | |
| Rendering and robot rules | |
| Applebot may render the content of your website within a browser. If javascript, CSS, and other resources are blocked via robots.txt, it may not be able to render the content properly. This includes XHR, JS, and CSS that the page might require. | |
| In order for Applebot to index the best content for the page, make sure that everything needed for a user to render the page is available to Applebot. Alternatively, make sure that the website renders cleanly, even if all of the resources are not available. This is often referred to as graceful degradation. | |
| Customizing indexing rules for Applebot | |
| Applebot supports robots meta tags in HTML documents. To specify robots rules in meta tags, put the tags in the <head> section of the document: | |
| <html><head> | |
| <meta name="robots" content="noindex"/> | |
| ... | |
| </head> | |
| <body>...</body> | |
| </html> | |
| <meta name="robots" content="nosnippet, noindex”> | |
| <meta name="robots" content=“noindex"> | |
| <meta name="robots" content=“nosnippet"> | |
| Applebot also supports the following directives: | |
| noindex: Applebot won't index this page, and it won't appear in Spotlight or Siri Suggestions. | |
| nosnippet: Applebot won't generate a description or web answer for the page. Any suggestions to visit this URL will only include the page's title. | |
| nofollow: Applebot won't follow any links on the page. | |
| none: Applebot won't index, snippet, or follow links on the page, as described above. | |
| all: Applebot provides the document for suggestions and snippets the contents so that a short description of the page can appear next to a representative image. Applebot may follow links on the page to provide more suggestions. | |
| To put multiple directives in a single meta tag, use a comma-separated list or multiple meta tags. | |
| Examples: | |
| <meta name="robots" content="nosnippet, noindex”> | |
| <meta name="robots" content=“noindex"> | |
| <meta name="robots" content=“nosnippet"> | |
| About search rankings | |
| Apple Search may take the following into account when ranking web search results: | |
| Aggregated user engagement with search results | |
| Relevancy and matching of search terms to webpage topics and content | |
| Number and quality of links from other pages on the web | |
| User location based signals (approximate data) | |
| Webpage design characteristics | |
| Search results may use the above factors with no (pre-determined) importance of ranking. Users of Search are subject to the privacy policy in Siri Suggestions, Search & Privacy. | |
| Contact us | |
| If you have questions or concerns, please contact us at applebot@apple.com. | |
| Information about products not manufactured by Apple, or independent websites not controlled or tested by Apple, is provided without recommendation or endorsement. Apple assumes no responsibility with regard to the selection, performance, or use of third-party websites or products. Apple makes no representations regarding third-party website accuracy or reliability. Contact the vendor for additional information. | |
| Published Date: May 08, 2024 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment