Skip to content

Instantly share code, notes, and snippets.

@danilvalov
Last active February 18, 2021 18:36
Show Gist options
  • Star 3 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save danilvalov/720966d26e99f06aed41 to your computer and use it in GitHub Desktop.
Save danilvalov/720966d26e99f06aed41 to your computer and use it in GitHub Desktop.
FlexGet Kinopoisk plugin
templates:
movies:
set:
path: /tmp/mnt/94E8B2B1E8B290CA/Torrents/download/DLNA/Movies
transmission:
host: ************
port: ****
username: *************
password: *************
email:
from: **********@**********.ru
to: **********@**********.ru
title: FlexGet Notification
smtp_host: *************
smtp_port: 25
smtp_username: *************
smtp_password: '*************'
tasks:
nnm-club:
rss:
url: http://nnm-club.me/forum/rss2.php?f=218&t=1&uk=*************
other_fields: [description]
ascii: yes
accept_all: yes
manipulate:
- title:
replace:
regexp: '[^\x00-\x80]+'
format: ''
- kinopoisk_id:
from: description
replace:
regexp: '(.|\n)*kinopoisk.ru/rating/([0-9]+)(.|\n)*'
format: '\2'
kinopoisk:
min_score: 7.0
min_votes: 1000
template:
- movies
from __future__ import unicode_literals, division, absolute_import
import urllib2, xmltodict
import logging
from flexget import plugin
from flexget.event import event
from flexget.plugins.filter.seen import FilterSeen
log = logging.getLogger('kinopoisk')
class FilterKinopoisk(FilterSeen):
"""
Kinopoisk filter
Using:
Add option to config.yml:
kinopoisk:
min_score: 7.0 # ignore movies with score below 7.0, default: 0
min_votes: 1000 # ignore movies with votes below 1000, default: 0
matching: 'strict' # ignore entries without kinopoisk_id, values: 'strict' or 'loose', default: 'strict'
scope: 'local' # use global or local storage, values: 'global' or 'local', default: 'global'
Please, replace values of '7.0' and '1000' to your
"""
schema = {
'oneOf': [
{
'type': 'object',
'properties': {
'min_score': {'type': 'number'},
'min_votes': {'type': 'integer'},
'matching': {'type': 'string', 'enum': ['strict', 'loose']},
'scope': {'type': 'string', 'enum': ['global', 'local']}
}
}
]
}
def __init__(self):
# remember and filter by these fields
self.fields = ['kinopoisk_id']
self.keyword = 'kinopoisk'
def is_number(self, s):
try:
float(s)
return True
except ValueError:
return False
@plugin.priority(0) # Make filter run after other filters, but before exists_movies
def on_task_filter(self, task, config):
if config is False:
return
# Reject all entries without kinopoisk id
if config.get('matching') != 'loose':
for entry in task.entries:
if 'kinopoisk_id' not in entry or not self.is_number(entry['kinopoisk_id']):
log.info('Rejecting %s because of missing movie kinopoisk id' % entry['title'])
entry.reject('Missing movie kinopoisk id')
# call super
super(FilterKinopoisk, self).on_task_filter(task, config.get('scope', True))
# check that two copies of a movie have not been accepted this run
kinopoisk_ids = set()
for entry in task.accepted:
if not self.is_number(entry['kinopoisk_id']):
entry.reject('Kinopoisk ID not found')
continue
if entry['kinopoisk_id'] in kinopoisk_ids and self.is_number(entry['kinopoisk_id']):
entry.reject('Already accepted once in task')
continue
else:
kinopoisk_ids.add(entry['kinopoisk_id'])
try:
page = urllib2.urlopen('http://rating.kinopoisk.ru/' + entry['kinopoisk_id'] + '.xml')
except urllib2.HTTPError:
entry.reject('Rating is not found')
continue
doc = page.read()
page.close()
data = xmltodict.parse(doc)
score = float(data['rating']['kp_rating']['#text'])
votes = float(data['rating']['kp_rating']['@num_vote'])
if 'min_score' in config:
if score <= 0:
entry.reject('Score is zero')
continue
if score < config['min_score']:
entry.reject('Score (%s) below minimum (%s)' % (score, config['min_score']))
continue
if 'min_votes' in config:
if votes < config['min_votes']:
entry.reject('Votes (%s) below minimum (%s)' % (votes, config['min_votes']))
continue
log.debug('Accepting %s' % (entry['title']))
@event('plugin.register')
def register_plugin():
plugin.register(FilterKinopoisk, 'kinopoisk', api_ver=2)
@nickson2006
Copy link

Hey.
I am trying to implement a plug-in in your config. It is not clear where to throw myself with the plugin file.
He throws in Flexget root juts error.

2016-02-26 20:43 CRITICAL manager [/tasks/nnm-club] The key kinopoisk is not valid here. Only known plugin names are valid keys.
2016-02-26 20:43 CRITICAL manager Failed to load config file: Did not pass schema validation.
Could not start manager: Did not pass schema validation.

@danilvalov
Copy link
Author

@nickson2006:

My flexget paths:

~/.flexget/config.yml
~/.flexget/plugins/kinopoisk.py
~/.flexget/templates/movies.template

You can delete this lines:

template:
      - movies

and use default template.

@danilvalov
Copy link
Author

Added handler for incorrect kinopoisk response.

@XBeg9
Copy link

XBeg9 commented Apr 28, 2016

@danilvalov how we login to nnmclub? where to set cookies or what's uk ?

@XBeg9
Copy link

XBeg9 commented Jul 17, 2016

@danilvalov it seems kinopoisk rating algorithm (parser) doesn't work anymore.

@githubbla
Copy link

githubbla commented Nov 5, 2016

@danilvalov how to find kinopoisk id from the page of torrent website, not from field of rss feed (where link in the field).

@danilvalov
Copy link
Author

@XBeg9
It works for me:

2016-12-07 19:11 VERBOSE  task          nnm-club        REJECTED: `/ Morgan (2016) BDRip [H.264/720p-LQ] [2.13GB]` by kinopoisk plugin because score (5.71) below minimum (7.0)
2016-12-07 19:11 VERBOSE  task          nnm-club        REJECTED: `/ The Shallows (2016) BDRip [1.46GB]` by kinopoisk plugin because score (6.272) below minimum (7.0)
2016-12-07 19:11 VERBOSE  task          nnm-club        REJECTED: `/ The BFG (2016) BDRip [2.19GB]` by kinopoisk plugin because score (6.129) below minimum (7.0)
2016-12-07 19:11 VERBOSE  task          nnm-club        REJECTED: `/ Run the Tide (2016) WEB-DL [H.264/720p-LQ] [3.12GB]` by kinopoisk plugin because score is zero
2016-12-07 19:11 VERBOSE  task          nnm-club        REJECTED: `/ Run the Tide (2016) WEB-DL [H.264/1080p-LQ] [3.43GB]` by kinopoisk plugin because already accepted once in task
2016-12-07 19:11 VERBOSE  task          nnm-club        REJECTED: `/ End of a Gun (2016) HDRip [1.36GB]` by kinopoisk plugin because score is zero
2016-12-07 19:11 VERBOSE  task          nnm-club        REJECTED: `/ Morgan (2016) BDRip [1.42GB]` by kinopoisk plugin because already accepted once in task
2016-12-07 19:11 VERBOSE  task          nnm-club        REJECTED: `- / Ben-Hur (2016) HDRip [Line] [2.05GB]` by kinopoisk plugin because score (5.892) below minimum (7.0)

@danilvalov
Copy link
Author

@githubroman
The following regexp finds kinopoisk ids in kinopoisk urls on any page:
https://gist.github.com/danilvalov/720966d26e99f06aed41#file-config-yml-L34

For example:
you have the following description:

<description>&lt;a href=&quot;https://nnmclub.to/forum/viewtopic.php?t=1080824&quot;&gt;тема на форуме&lt;/a&gt;&lt;br&gt;&lt;span style=&quot;text-align: center; display: block;&quot;&gt;&lt;span style=&quot;color: Indigo&quot;&gt;&lt;span style=&quot;font-size: 20px; line-height: normal&quot;&gt;&lt;span style=&quot;font-weight: bold&quot;&gt; Морган / Morgan (2016) BDRip [H.264/720p-LQ]&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&quot;text-align: center; display: block;&quot;&gt;&lt;span style=&quot;color: gray&quot;&gt;&lt;span style=&quot;font-weight: bold&quot;&gt;«Don't let it out»&lt;/span&gt;&lt;/span&gt;&lt;/span&gt; &lt;hr /&gt;&lt;img src=&quot;{ASSETS}/forum/image.php?link=http://s020.radikal.ru/i713/1612/4b/0b3129627bb5.jpg&quot;&gt;&lt;span style=&quot;font-weight: bold&quot;&gt;Производство:&lt;/span&gt; США / Scott Free Productions&lt;br /&gt;&lt;span style=&quot;font-weight: bold&quot;&gt;Жанр:&lt;/span&gt; Ужасы, Фантастика, Триллер, Детектив&lt;br /&gt;&lt;br /&gt;&lt;span style=&quot;font-weight: bold&quot;&gt;Режиссер:&lt;/span&gt; Люк Скотт&lt;br /&gt;&lt;span style=&quot;font-weight: bold&quot;&gt;Актеры:&lt;/span&gt; Кейт Мара, Аня Тейлор-Джой, Роуз Лесли, Майкл Йар, Тоби Джонс, Крис Салливан, Бойд Холбрук, Винетт Робинсон, Мишель Йео, Брайан Кокс&lt;br /&gt;&lt;br /&gt;&lt;span style=&quot;font-weight: bold&quot;&gt;Описание:&lt;/span&gt;&lt;br /&gt;Сотрудница по устранению аварий отправляется в отдаленный секретный филиал, где должна расследовать и оценить потери от ужасной катастрофы. На месте она узнает, что случившееся было вызвано, казалось бы, невинным «человеком», чье существование само по себе является опасной тайной.&lt;br /&gt;&lt;br /&gt;&lt;span id=&quot;kpjsrch&quot;&gt;&lt;/span&gt; &lt;a id=&quot;kp_id&quot; href=&quot;http://www.kinopoisk.ru/film/906340/&quot; target=&quot;_blank&quot; rel=&quot;nofollow&quot;&gt;&lt;img src=&quot;https://www.kinopoisk.ru/rating/906340.gif&quot;&gt;&lt;/a&gt; &lt;a id=&quot;imdb_id&quot; href=&quot;http://www.imdb.com/title/tt4520364/&quot; target=&quot;_blank&quot; rel=&quot;nofollow&quot;&gt;&lt;img src=&quot;http://imdb.snick.ru/ratefor/02/tt4520364.png&quot;&gt;&lt;/a&gt;&lt;br /&gt;&lt;span style=&quot;font-size: 11px; line-height: normal&quot;&gt;&lt;span style=&quot;font-weight: bold&quot;&gt;Возраст:&lt;/span&gt; &lt;span style=&quot;color: #FF0000&quot;&gt;&lt;span style=&quot;font-weight: bold&quot;&gt;18+&lt;/span&gt;&lt;/span&gt; &lt;span style=&quot;color: #CC0000&quot;&gt;(зрителям, достигшим 18 лет. запрещено для детей)&lt;/span&gt;&lt;/span&gt;&lt;br /&gt;&lt;span style=&quot;font-size: 11px; line-height: normal&quot;&gt;&lt;span style=&quot;font-weight: bold&quot;&gt;Рейтинг MPAA:&lt;/span&gt; &lt;span style=&quot;color: #FF6000&quot;&gt;&lt;span style=&quot;font-weight: bold&quot;&gt;R&lt;/span&gt; (лицам до 17 лет обязательно присутствие взрослого)&lt;/span&gt;&lt;/span&gt;&lt;br /&gt;&lt;span style=&quot;font-size: 11px; line-height: normal&quot;&gt;&lt;span style=&quot;font-weight: bold&quot;&gt;Дата мировой премьеры:&lt;/span&gt; 1 сентября 2016&lt;/span&gt;&lt;br /&gt;&lt;span style=&quot;font-size: 11px; line-height: normal&quot;&gt;&lt;span style=&quot;font-weight: bold&quot;&gt;Дата российской премьеры:&lt;/span&gt; 8 сентября 2016, «Двадцатый Век Фокс СНГ»&lt;/span&gt;&lt;br /&gt;&lt;br /&gt;&lt;span style=&quot;font-weight: bold&quot;&gt;Релиз от:&lt;/span&gt; &lt;img src=&quot;{ASSETS}/forum/image.php?link=http://s020.radikal.ru/i712/1503/80/49b7bc4e3261.gif&quot;&gt; &lt;span style=&quot;color: blue&quot;&gt;NNMClub&lt;/span&gt;&lt;br /&gt;&lt;br /&gt;&lt;span style=&quot;font-weight: bold&quot;&gt;Продолжительность:&lt;/span&gt; 01:31:52&lt;br /&gt;&lt;span style=&quot;font-weight: bold&quot;&gt;Качество видео:&lt;/span&gt; BDRip &lt;a href=&quot;http://sendfile.su/1288607&quot; rel=&quot;nofollow&quot; class=&quot;postLink&quot;&gt;&lt;img src=&quot;{ASSETS}/forum/image.php?link=http://nnmclub.to/forum/images/channel/sample_light_nnm.png&quot;&gt;&lt;/a&gt;&lt;br /&gt;&lt;span style=&quot;font-weight: bold&quot;&gt;Перевод:&lt;/span&gt; Дублированный (iTunes)&lt;br /&gt;&lt;span style=&quot;font-weight: bold&quot;&gt;Вид субтитров:&lt;/span&gt; Вшитые отключаемые&lt;br /&gt;&lt;span style=&quot;font-weight: bold&quot;&gt;Язык субтитров:&lt;/span&gt; Русский (форсированные на иностранные надписи)&lt;br /&gt;&lt;br /&gt;&lt;span style=&quot;font-weight: bold&quot;&gt;Видео:&lt;/span&gt; H.264/AVC, 1280x536, 2938 kbps, 23.976 fps&lt;br /&gt;&lt;span style=&quot;font-weight: bold&quot;&gt;Аудио:&lt;/span&gt; AC3, 6 ch, 384 Kbps - Русский &lt;/a&gt;&lt;div class=&quot;clear&quot;&gt;&lt;/div&gt; &lt;/a&gt;&lt;div class=&quot;clear&quot;&gt;&lt;/div&gt; &lt;div class=&quot;clear&quot;&gt;&lt;/div&gt;&lt;span style=&quot;text-align: center; display: block;&quot;&gt;&lt;span style=&quot;font-weight: bold&quot;&gt;Скриншоты:&lt;/span&gt;&lt;br /&gt;&lt;a href=&quot;http://i5.imageban.ru/out/2016/12/07/d35e0e0ae99ca77b3ebdc69e77ec85cf.png&quot; class=&quot;highslide&quot; onclick=&quot;return hs.expand(this,{slideshowGroup: '8668771'})&quot; rel=&quot;nofollow&quot;&gt;&lt;img src=&quot;{ASSETS}/forum/image.php?link=http://i4.imageban.ru/out/2016/12/07/4fc737ff3ff2f5e21f4fdc8e3027e635.png&quot;&gt;&lt;/a&gt;  &lt;a href=&quot;http://i2.imageban.ru/out/2016/12/07/7637dcf726f41bd78ef402e4ef8b78c9.png&quot; class=&quot;highslide&quot; onclick=&quot;return hs.expand(this,{slideshowGroup: '8668771'})&quot; rel=&quot;nofollow&quot;&gt;&lt;img src=&quot;{ASSETS}/forum/image.php?link=http://i3.imageban.ru/out/2016/12/07/a3af40b6b0117e8618196ce7ec2f6056.png&quot;&gt;&lt;/a&gt;  &lt;a href=&quot;http://i5.imageban.ru/out/2016/12/07/618b741b4b80ac0c1d462339ed545cf0.png&quot; class=&quot;highslide&quot; onclick=&quot;return hs.expand(this,{slideshowGroup: '8668771'})&quot; rel=&quot;nofollow&quot;&gt;&lt;img src=&quot;{ASSETS}/forum/image.php?link=http://i1.imageban.ru/out/2016/12/07/163e19955ccd5ff8e8773cddb5f1c29e.png&quot;&gt;&lt;/a&gt;  &lt;a href=&quot;http://i3.imageban.ru/out/2016/12/07/8db0d0a44c1afe42868e6a84f4341f4c.png&quot; class=&quot;highslide&quot; onclick=&quot;return hs.expand(this,{slideshowGroup: '8668771'})&quot; rel=&quot;nofollow&quot;&gt;&lt;img src=&quot;{ASSETS}/forum/image.php?link=http://i3.imageban.ru/out/2016/12/07/85bec5c826a2796e7378cd5d37ea22fe.png&quot;&gt;&lt;/a&gt;&lt;br /&gt;&lt;/span&gt;&lt;br /&gt;&lt;span style=&quot;text-align: center; display: block;&quot;&gt;&lt;a href=&quot;http://nnmclub.to/?q=Morgan+2016&amp;amp;w=title&quot; rel=&quot;nofollow&quot; class=&quot;postLink&quot;&gt;&lt;span style=&quot;font-size: 16px; line-height: normal&quot;&gt;&lt;span style=&quot;color: #800000&quot;&gt;&lt;span style=&quot;font-weight: bold&quot;&gt;Все одноименные релизы в Клубе&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/a&gt;&lt;/span&gt;&lt;br /&gt;&lt;span style=&quot;font-size: 12px; line-height: normal&quot;&gt;&lt;span style=&quot;font-weight: bold&quot;&gt;Время раздачи:&lt;/span&gt; 24/7 до первых 10-и скачавших&lt;/span&gt;</description>

and we can see the following part:

... href=&quot;http://www.kinopoisk.ru/film/906340/&quot; ...

The script find this url by regexp and get the kinopoisk id.
You can use this manipulate for any pages with kinopoisk urls. Or you can rewrite the regexp to get the kinopoisk id from another data types.

@danilvalov
Copy link
Author

@XBeg9

how we login to nnmclub? where to set cookies or what's uk ?
uk is the passkey from the update profile page:
https://nnmclub.to/forum/profile.php?mode=editprofile

@advokatb
Copy link

advokatb commented Apr 23, 2018

2018-04-22 21:21 CRITICAL task          nnm-club        BUG: Unhandled error in plugin kinopoisk: not well-formed (invalid token): line 1, column 0
Traceback (most recent call last):
  File "d:\program\python\python27\lib\site-packages\flexget\task.py", line 486, in __run_plugin
    return method(*args, **kwargs)
  File "d:\program\python\python27\lib\site-packages\flexget\event.py", line 23, in __call__
    return self.func(*args, **kwargs)
  File "C:\Users\222\flexget\plugins\kinopoisk.py", line 90, in on_task_filter
    data = xmltodict.parse(doc)
  File "d:\program\python\python27\lib\site-packages\xmltodict.py", line 330, in parse
    parser.Parse(xml_input, True)
ExpatError: not well-formed (invalid token): line 1, column 0

Как пофиксить? Спасибо.

@victornest
Copy link

victornest commented Feb 18, 2021

2018-04-22 21:21 CRITICAL task          nnm-club        BUG: Unhandled error in plugin kinopoisk: not well-formed (invalid token): line 1, column 0
Traceback (most recent call last):
  File "d:\program\python\python27\lib\site-packages\flexget\task.py", line 486, in __run_plugin
    return method(*args, **kwargs)
  File "d:\program\python\python27\lib\site-packages\flexget\event.py", line 23, in __call__
    return self.func(*args, **kwargs)
  File "C:\Users\222\flexget\plugins\kinopoisk.py", line 90, in on_task_filter
    data = xmltodict.parse(doc)
  File "d:\program\python\python27\lib\site-packages\xmltodict.py", line 330, in parse
    parser.Parse(xml_input, True)
ExpatError: not well-formed (invalid token): line 1, column 0

Как пофиксить? Спасибо.

Looks like sometimes it gives reponse in gzip format. It can be fixed like that:

doc = page.read()
page.close()
try:
    data = xmltodict.parse(doc)
except ExpatError:
    compressed_data = io.BytesIO(doc)
    for compressed_data_item in gzip.GzipFile(fileobj=compressed_data):
        data = xmltodict.parse(compressed_data_item)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment