Skip to content

Instantly share code, notes, and snippets.

Avatar

Sanhe Hu MacHu-GWU

  • Amazon Web Service Data Lab
  • Maryland, USA
View GitHub Profile
@MacHu-GWU
MacHu-GWU / mongocrawl.py
Last active October 3, 2016 18:27
A MongoDB backed, width first web crawler framework.
View mongocrawl.py
#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""
**中文文档**
mongocrawl是一个使用MongoDB作为数据持久层, 广度优先的定向爬虫框架。非常适合于
数据资源所在的Url是树状层级结构的爬虫项目。
View outbrain-database.rst

Define the table

Because each document may have multiple category, topic and entity. If we only use the highest confident label, then the complexity can be greatly reduced. After that, document vs category, topic and entity would be many-to-one relationship.

View homework1.py
#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""
2017-01-26
You have to under stand these:
basic concept:
@MacHu-GWU
MacHu-GWU / inspect_mate.py
Created May 9, 2017 20:34
``inspect_mate`` provides more methods to get information about class attribute than the standard library ``inspect``.
View inspect_mate.py
#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""
``inspect_mate`` provides more methods to get information about class attribute
than the standard library ``inspect``.
This module is Python2/3 compatible, tested under Py2.7, 3.3, 3.4, 3.5, 3.6.
Includes tester function to check:
@MacHu-GWU
MacHu-GWU / upsert.py
Created June 9, 2017 18:21 — forked from timtadh/upsert.py
How to compile an INSERT ... ON DUPLICATE KEY UPDATE with SQL Alchemy with support for a bulk insert.
View upsert.py
#!/usr/bin/env python
# Copyright (c) 2012, Tim Henderson
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are
# met:
#
# - Redistributions of source code must retain the above copyright notice,
# this list of conditions and the following disclaimer.
@MacHu-GWU
MacHu-GWU / Keymap-Template.rst
Last active August 16, 2017 20:18
A template for keymap documentation for TextEditor/IDLE
@MacHu-GWU
MacHu-GWU / PyCharm-Shortcuts-and-Keymap.rst
Created August 16, 2017 20:24
PyCharm Shortcuts and Keymap
View PyCharm-Shortcuts-and-Keymap.rst