codervince

rpsider.py

1.   for i, r in enumerate(response.xpath("//table/tbody")):
                GET following items from grid table
               for i, r in enumerate(response.xpath("//table/tbody")):
                #tds - blank horsenumber, lbw, horsename country sp (td span a), age, carrierWt, OR, TS , OPR,  RATED 
                #if * then not logged in!
                position
                lbw
                horse_id

codervince

#in cron file
* * * * * /Users/vmac/Documents/PROGRAMMING/PY/scrapy/NEWHKODDS/v3/HKOdds/test.sh

#error
Traceback (most recent call last):
  File "/Users/vmac/Documents/PROGRAMMING/PY/scrapy/NEWHKODDS/v3/HKOdds/HKOddsCollector.py", line 210, in <module>
    collector = HKOddsCollector()
  File "/Users/vmac/Documents/PROGRAMMING/PY/scrapy/NEWHKODDS/v3/HKOdds/HKOddsCollector.py", line 30, in __init__
    self.db = Database()
  File "/Users/vmac/Documents/PROGRAMMING/PY/scrapy/NEWHKODDS/v3/HKOdds/database.py", line 24, in __init__

codervince

WITH t1 as
(
SELECT distinct
  hk_odds.racenumber as thisracenumber, 
  hk_odds.horsenumber as thishorsenumber,
    (select winodds from hk_odds o2 where o2.racenumber =  hk_odds.racenumber and o2.horsenumber = hk_odds.horsenumber order by updatedate asc LIMIT 1) as openingprice,
   (select updatedate from hk_odds o2 where o2.racenumber =  hk_odds.racenumber and o2.horsenumber = hk_odds.horsenumber order by updatedate desc LIMIT 1) as currenttime,
  (select winodds from hk_odds o2 where o2.racenumber =  hk_odds.racenumber and o2.horsenumber = hk_odds.horsenumber order by updatedate desc LIMIT 1) as currentodds,
  (select winodds from hk_odds o2 where o2.racenumber =  hk_odds.racenumber and o2.horsenumber = hk_odds.horsenumber order by updatedate desc LIMIT 1 OFFSET 1) as L1odds,
  (select winodds from hk_odds o2 where o2.racenumber =  hk_odds.racenumber and o2.horsenumber = hk_odds.horsenumber order by updatedate desc LIMIT 1 OFFSET 2) as L2odds,

codervince

#fields for rpresults
http://www.racingpost.com/horses/result_home.sd?race_id=620250&r_date=2015-03-24&popup=yes#results_top_tabs=re_&results_bottom_tabs=ANALYSIS

TOP
racedate (URL)
racecourse
rtime e.g. 2.10 
racename

BOTTOM

codervince

https://github.com/codervince/rpostraceday

Task:
1. if any page (esp. pedigreequery.com) returns 404 ie no item fields matching URL, still collect and return item as before
2. RacedayItemLoader should TakeFirst() for [] single element lists or Join for multiple element lists - matching pipeline definitions

codervince

HKResults project

Phase 2 
PRELIMINARIES:
take hkjc_spider.py from andrey-chabrov/racingpost
new repo https://github.com/codervince/hkjcresults.git

Add following fields, e.g.
http://racing.hkjc.com/racing/Info/Meeting/Results/English/Local/20150524/ST/11
parse_race

codervince

#1 

The data from getdata() is in the data.pkl file. 
First create the users from tipsters = np.array(<USER.NAME>)
Then upload data from data.pkl, one row per user to the 
TipsterPerformance table.
Note the racedate for the INIT data is say AUG 1 2015

All the SQLAlchemy code should be in a separate file.
TipsterPerformance replaces UserPerformance.

codervince

http://www.hkjc.com/english/racing/OtherHorse.asp?HorseNo=N159

http://www.hkjc.com/english/racing/horse.asp?horseno=M327

There are two variants above plus a variant for retired horses



Each horse has a code a050 to t999

codervince

##file 1
http://streaming.hkjc.edgesuite.net/mobile/audio/audio_eng_20150712.mp3

codervince

def convert_to_bin_places(seq):
    rtn = ""
    if '>' in seq:
        '''remove > '''
        seq = seq.split('>')
    rtn = list(map(lambda x: '1' if x != '' and x == '1' else '0', seq))
    return "".join(rtn)


def convert_to_fs_places(seq):