You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Instantly share code, notes, and snippets.
Carl Aiau
carlaiau
@Sunobi Ninja. Solar Hackin. CompSci PGDipSci @ Otago Uni. Getting pulled into Web3
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
These both work great if performed during our initial account creation (Prior to a billing usage profile getting assigned for the account), but the kWh -> $ calculation bugs out for subsequent requests after this.
Our user journey requires the creation of an account with either their yearly kWh or bill $ and us working out the other value. From this given result, we create an average usage billing profile for this account based on their inputted kWh amount, or the calculated kWh amount if they entered a bill instead and store the other values on our own CRM.
This billing profile is required for us to run scenarios against their previous usage, and the future solar installation.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
All scrapers within this folder are for scraping MyWebGrocer powered platforms.
mwg sites reside on the mywebgrocer domain, where as Curbside Express, Harris Teeter and Shoprite, have their own domains.
The three external sites have a JSON API layer, so are much quicker to scrape than the Xpath sraping of the mywebgrocer sites. There are similarities in the structure of the sites, but not enough to allow one spider built for multi domain.
The complete scraping, "middleware" and "post-processing" is done via calling the python files in the root proceeded with "run_"
The complete scraping, "middleware" and "post-processing" is done via calling the master.sh bash script
examining this file, along with the three python files that are called from within it, will give you an underdtanding of how this spider works.
I have chosen to create seperate external files for zones, stores, warehouses, categories, and products, rather than keeping the data in memory.
But we could amend the scrapers to utilize an item system.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters