Features¶

If you have needed to parse dates before, you probably have used the date parser of the dateutil module. We built this library on top of it, adding a few features:

• dateparser support dates in languages other than English
• in fact, it can detect the language automatically
• it can give you the date for text like: '1 min ago', '2 weeks ago', '3 months, 1 weeks and 1 day ago', etc

The goal is to support the common date formats used in websites all around the world.

Limitations¶

DateParser currently tries hard to get the date information right (year, month and day), but it has limited support for parsing time (hours, minutes and seconds).

Note about language detection¶

As it is now, an instance of DateDataParser by default assumes that all of the dates fed to it will be in the same language.

So, it will keep trying to detect the possible languages until it reduces it to only one possibility. When it does, it will just assume that for the next dates and won’t try to execute the language detection again:

>>> ddp = DateDataParser()
>>> ddp.get_date_data('1 minuto atrás')
{'date_obj': datetime.datetime(2014, 8, 20, 21, 1, 42, 590596), 'period': u'day'}
>>> ddp.get_date_data('13 Agosto 2014')
{'date_obj': datetime.datetime(2014, 8, 13, 0, 0), 'period': 'day'}
>>> ddp.get_date_data('13 Marzo 2014')
{'date_obj': datetime.datetime(2014, 3, 13, 0, 0), 'period': 'day'}
>>> ddp.get_date_data('13 Maio 2014')
Traceback (most recent call last):
...
dateparser.date_parser.LanguageWasNotSeenBeforeError

If you want it to redetect the language every time, you can use a custom date_parser, like so:

>>> ddp = DateDataParser()
>>> from dateparser.date_parser import DateParser
>>> ddp.date_parser = DateParser(allow_redetect_language=True)
>>> ddp.get_date_data('1 minuto atrás')
{'date_obj': datetime.datetime(2014, 8, 20, 21, 1, 42, 590596), 'period': u'day'}
>>> ddp.get_date_data('13 Agosto 2014')
{'date_obj': datetime.datetime(2014, 8, 13, 0, 0), 'period': 'day'}
>>> ddp.get_date_data('13 Marzo 2014')
{'date_obj': datetime.datetime(2014, 3, 13, 0, 0), 'period': 'day'}
>>> ddp.get_date_data('13 Maio 2014')
{'date_obj': datetime.datetime(2014, 5, 13, 0, 0), 'period': 'day'}

How to use in a Scrapy Cloud project¶

To use in Scrapy Cloud, first you need to build an egg for the library.

Clone the repo and inside its directory, run the command:

python setup.py bdist_egg

After that, you can upload the egg using Scrapy Cloud’s Dashboard interface, or you can use shubc command and do:

shubc eggs-add <YOUR_PROJECT_ID> dist/dateparser-0.1-py2.7.egg

Types of Contributions¶

Get Started!¶

Ready to contribute? Here’s how to set up dateparser for local development.

1. Fork the dateparser repo on GitHub.

2. Clone your fork locally:

   $ git clone git@github.com:your_name_here/dateparser.git
   

3. Install your local copy into a virtualenv. Assuming you have virtualenvwrapper installed, this is how you set up your fork for local development:

   $ mkvirtualenv dateparser
   $ cd dateparser/
   $ python setup.py develop
   

4. Create a branch for local development:

   $ git checkout -b name-of-your-bugfix-or-feature
   

   Now you can make your changes locally.

5. When you’re done making changes, check that your changes pass flake8 and the tests, including testing other Python versions with tox:

   $ pip install -r tests/requirements.txt # install test dependencies
   $ flake8 dateparser tests
   $ nosetests
   $ tox
   

   To get flake8 and tox, just pip install them into your virtualenv.

6. Commit your changes and push your branch to GitHub:

   $ git add .
   $ git commit -m "Your detailed description of your changes."
   $ git push origin name-of-your-bugfix-or-feature
   

7. Submit a pull request through the GitHub website.

Pull Request Guidelines¶

Before you submit a pull request, check that it meets these guidelines:

1. The pull request should include tests.
2. If the pull request adds functionality, the docs should be updated. Put your new functionality into a function with a docstring, and add the feature to the list in README.rst.
3. Check https://travis-ci.org/scrapinghub/dateparser/pull_requests and make sure that the tests pass for all supported Python versions.

Tips¶

To run a subset of tests:

$ nosetests tests.test_date_parser

Development Lead¶

• Claudio Salazar <claudio@scrapinghub.com>
• Elias Dorneles <elias@scrapinghub.com>
• Eugene Amirov <eugene@scrapinghub.com>

Contributors¶

Shuai Lin, Ismael Carnales and other folks from Scrapinghub staff.