http_parser.http.NoMoreData: Can't parse headers

I try to fetch location header from a 301 response. Sometime it crashs.

with some website, the socketstream don't feed the buffer. The parser raise NoMoreData. Same url with curl cli works very well.

@athoune

@athoune can you provide me a link to these websites? Or maybe just a trace if you by setting restkit logs to debug: restkit.send_logging("debug")?

Hi, I've found this occasionally occurs when running our Buildbot unit tests with CouchDBKit. It seems somewhat intermittent, so it's been difficult to track down. The versions I've tried are:

http-parser 0.7.4, http-parser 0.7.5
restkit 4.1.3

On my local machine, I'm using:
http-parser 0.7.1
restkit 3.3.2
and this combination seems to work ok.

set_logging. The bug only appears "under pressure", one url in a loop just raise

File "/Library/Python/2.6/site-packages/restkit/client.py", line 304, in perform
conn = self.get_connection(request)
File "/Library/Python/2.6/site-packages/restkit/client.py", line 193, in get_connection
extra_headers=extra_headers, **self.ssl_args)
File "/Library/Python/2.6/site-packages/socketpool/pool.py", line 198, in get
raise last_error
MaxConnectionsError: 1500

Traceback (most recent call last):
File "test.py", line 9, in
r = request('http://appsactifry.tevolys.com/sites/default/files/imagecache/recipe_detail_high/recipe_photos/348.jpg', pool=pool)
File "/Library/Python/2.6/site-packages/restkit/init.py", line 107, in request
headers=headers)
File "/Library/Python/2.6/site-packages/restkit/client.py", line 422, in request
return self.perform(request)
File "/Library/Python/2.6/site-packages/restkit/client.py", line 304, in perform
conn = self.get_connection(request)
File "/Library/Python/2.6/site-packages/restkit/client.py", line 193, in get_connection
extra_headers=extra_headers, **self.ssl_args)
File "/Library/Python/2.6/site-packages/socketpool/pool.py", line 198, in get
raise last_error
socketpool.pool.MaxConnectionsError: 1500

The test :
from restkit import *
set_logging("debug")

from socketpool import ConnectionPool

pool = ConnectionPool(factory=Connection, max_conn=1500)

for a in range(10000):
r = request('http://appsactifry.tevolys.com/sites/default/files/imagecache/recipe_detail_high/recipe_photos/348.jpg', pool=pool)
r.headers
print '.',

I don't understand why the pool is not large enough. This test is sequential.

The real use case is an import script. Fetch data from one site, put its to another site. Some data (pictures) are streamed in order to avoid disk access or fulling memory. To limit crash, parts of the fetch are done with urllib3. I'll try to provide a script to raise the error, but it seems to be hard.

Is there any progress on that? I am getting that error occasionally and I am interested in it being solved.

To release the connection you need to read the body if needed. or since 15431c5 you can skip the body using the skip_body() method of the response. Let me know if it works for you

remove the max_conn setting. fix #104

@benoitc

@benoitc it still happens quite frequently. with latest versions currently available. (I'm using restkit because of couchdbkit).

The NoMoreData("Can't parse headers")

@pigmej

@pigmej what is your couchdb version?

@benoitc

@benoitc happens on BigCouch 0.4, BigCouch master branch, CouchDB 1.2

@benoitc

@benoitc I'm having the same issue here:

NoMoreData: Can't parse headers (Full traceback: http://pastebin.com/QuHddbmn)

This error occurs more than often.
I'm using couchdbkit 0.6.4, with restkit 4.2.1 and http_parser 0.8.1

http://www.bpaste.net/show/iF8v5260Zky4nd14R4TD/ should give more information maybe (just a random guess, cant test it here)

That's basically what I did trying to troubleshoot this. I could never reproduce in a meaningful way. After reinstalling/updating all of the libraries used by restkit, this issue went away for me, and I've not seen it in production since.

I still have this problem, even on fully upgraded restkit, http-parser etc. So I would not tell that it went away with updates :/.

-----Original Message-----
From: Wendall Cada notifications@github.com
Date: Thu, 14 Mar 2013 09:21:26
To: benoitc/restkitrestkit@noreply.github.com
Reply-To: benoitc/restkit reply@reply.github.com
Cc: Jędrzej Nowakpigmej@gmail.com
Subject: Re: [restkit] http_parser.http.NoMoreData: Can't parse headers (#104)

That's basically what I did trying to troubleshoot this. I could never reproduce in a meaningful way. After reinstalling/updating all of the libraries used by restkit, this issue went away for me, and I've not seen it in production since.

Reply to this email directly or view it on GitHub:
#104 (comment)

@benoitc

@benoitc it happens on production. Not an option to provide URL to internal
systems obviously.

It happens when using haproxy with bigcouch (haproxy doesn't reports broken
connections).

It's pretty loaded system. It happens time to time (in like 1% of requests
max I think). And it works perfectly fine with requests backend instead.

Pozdrawiam
Jędrzej Nowak

On Fri, Mar 15, 2013 at 3:10 PM, Benoit Chesneau
notifications@github.comwrote:

@pigmej https://github.com/pigmej when does that happen? Any way I can
reproduce? Maybe you can provide an url we can test on?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/104#issuecomment-14962535
.

New observations. Happens on both 2.6 and 2.7 python versions.

happens both for python and C version of parser.

@pigmej

In my system, it happens quite frequently.
@pigmej I am not using Haproxy so that should not be an error factor

@pigmej

@pigmej can you tell me the number of requests when it happens? (in private if you prefer)

@dirkmoors is this oin a loaded system too?

I would like to write a test to make sure I fix it for this time.

@benoitc

@benoitc
Doesn't matter, sometimes when there are 100+ / sec sometimes on one request in 5 seconds.

Generally what I was abble to debug, when it happens I have 0 data read from buffer (see patch at bottom).

diff --git a/http_parser/http.py b/http_parser/http.py
index 17ade32..95695f1 100644
--- a/http_parser/http.py
+++ b/http_parser/http.py
@@ -10,6 +10,8 @@ try:
 except ImportError:
     from http_parser.pyparser import HttpParser

+import time
+
 from http_parser.reader import HttpBodyReader
 from http_parser.util import status_reasons, bytes_to_str

@@ -45,6 +47,8 @@ class HttpStream(object):
         """
         self.parser = HttpParser(kind=kind, decompress=decompress)
         self.stream = stream
+        self._tmp_data = []
+        self._tmp_recv = []

     def _check_headers_complete(self):
         if self.parser.is_headers_complete():
@@ -56,7 +60,13 @@ class HttpStream(object):
             except StopIteration:
                 if self.parser.is_headers_complete():
                     return
-                raise NoMoreData("Can't parse headers")
+                t = time.time()
+                with open('/tmp/nomoredata_%s_data' % t, 'wb') as f:
+                    f.write(''.join(self._tmp_data))
+                with open('/tmp/nomoredata_%s_recv' % t, 'wb') as f:
+                    f.write('\n'.join(self._tmp_recv))
+
+                raise NoMoreData("Can't parse headers %s" % t)

             if self.parser.is_headers_complete():
                 return
@@ -204,6 +214,9 @@ class HttpStream(object):
             raise ParserError("nparsed != recved (%s != %s) [%s]" % (nparsed,
                 recved, bytes_to_str(to_parse)))

+        self._tmp_data.append(to_parse)
+        self._tmp_recv.append(str(recved))
+
         if recved == 0:
             raise StopIteration

@benoitc

@benoitc No, not necessarily, even when the server is completely idle (development server), the error occurs quite frequently

are you using the latest version of socketpool?

@benoitc

@benoitc My version for socketpool is 0.5.2.

@benoitc

@benoitc I have @0.5.2@ too.

And I have similar observations to @dirkmoors, it happens "randomly" but frequently. I would say, it happens even more on non loaded system that on loaded one (I might be wrong there.)

Some more information (something that I noticed while looking further into the error trace):

I'm using django_couchdb_utils (https://github.com/theju/django-couchdb-utils).
When I want to validate whether a user is logged in, I run the following function:

def is_logged_in(request):  
    if request.method != "POST":
        return render405(request)

    user = get_user(request)

    data = {
        "is_logged_in"  : user.is_authenticated() if user else False,
        "username"      : getattr(user, "username", None)        
    }

    request.session.load()

    return render_view(request, data=data, content_type="application/json", addCSRF=True)

As you can see, that calls "get_user(request)" which is implemented like this in django:

def get_user(request):
    from django.contrib.auth.models import AnonymousUser
    try:
        user_id = request.session[SESSION_KEY]
        backend_path = request.session[BACKEND_SESSION_KEY]
        backend = load_backend(backend_path)
        user = backend.get_user(user_id) or AnonymousUser()
    except KeyError:
        user = AnonymousUser()
    return user

The error is thrown on the first statement inside the try-except: "user_id = request.session[SESSION_KEY]":

...
File "/usr/local/lib/python2.7/dist-packages/django/contrib/auth/init.py", line 98, in get_user
user_id = request.session[SESSION_KEY]
File "/usr/local/lib/python2.7/dist-packages/django/contrib/sessions/backends/base.py", line 39, in getitem
return self._session[key]
File "/usr/local/lib/python2.7/dist-packages/django/contrib/sessions/backends/base.py", line 165, in _get_session
self._session_cache = self.load()
File "/usr/local/lib/python2.7/dist-packages/django_couchdb_utils/sessions/couchdb.py", line 21, in load
session = Session.get_session(self.session_key)
File "/usr/local/lib/python2.7/dist-packages/django_couchdb_utils/sessions/models.py", line 20, in get_session
return r.first()
File "/usr/local/lib/python2.7/dist-packages/couchdbkit/client.py", line 957, in first
return list(self)[0]
File "/usr/local/lib/python2.7/dist-packages/couchdbkit/client.py", line 1062, in len
return self.count()
File "/usr/local/lib/python2.7/dist-packages/couchdbkit/client.py", line 988, in count
self._fetch_if_needed()
File "/usr/local/lib/python2.7/dist-packages/couchdbkit/client.py", line 1019, in _fetch_if_needed
self.fetch()
File "/usr/local/lib/python2.7/dist-packages/couchdbkit/client.py", line 1001, in fetch
self._result_cache = self.fetch_raw().json_body
File "/usr/local/lib/python2.7/dist-packages/couchdbkit/client.py", line 1015, in fetch_raw
return self._fetch(self._arg, self.params)
File "/usr/local/lib/python2.7/dist-packages/couchdbkit/client.py", line 697, in raw_view
return self.res.get(view_path, *_params)
File "/usr/local/lib/python2.7/dist-packages/restkit/resource.py", line 114, in get
params_dict=params_dict, *_params)
File "/usr/local/lib/python2.7/dist-packages/couchdbkit/resource.py", line 111, in request
payload=payload, headers=headers, **params)
File "/usr/local/lib/python2.7/dist-packages/restkit/resource.py", line 190, in request
headers=self.make_headers(headers))
File "/usr/local/lib/python2.7/dist-packages/restkit/client.py", line 413, in request
return self.perform(request)
File "/usr/local/lib/python2.7/dist-packages/restkit/client.py", line 345, in perform
return self.get_response(request, conn)
File "/usr/local/lib/python2.7/dist-packages/restkit/client.py", line 450, in get_response
location = p.headers().get('location')
File "/usr/local/lib/python2.7/dist-packages/http_parser/http.py", line 135, in headers
self._check_headers_complete()
File "/usr/local/lib/python2.7/dist-packages/http_parser/http.py", line 59, in _check_headers_complete
raise NoMoreData("Can't parse headers")
NoMoreData: Can't parse headers

I was having this issue quite regularly. I updated all my packages with pip and haven't experienced it since

should be fixed with the latest version of http-parser.

I'm still seeing this, albeit very sporadically. How can I provide details for debugging? A stack trace doesn't seem very likely to be helpful, but here one is anyway.

https://gist.github.com/isolationism/7ce33574b3d1cc415477

I'm getting the same error when requesting a large number of documents in a loop by using couchkit. I checked the number of loops until the error occurs. Here are the numbers (each line is one run):

11312 & 11313
11161 & 11162
11746 & 11747
14825 & 14826
11229 & 11230
15297 & 15298

The error always occurs in pairs and disappears just after. I have just a pool of ca. 20000 documents so I can't tell if the error will occur again after 11000 - 16000 documents.

I'm using the lasted version of socketpool and http-parser. I develop with python 2.7

sounds like a bug in socketpool... will see what ic ould do about that.

Navigation Menu

Repository navigation

http_parser.http.NoMoreData: Can't parse headers #104

3 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

http_parser.http.NoMoreData: Can't parse headers #104

Description

Activity

athoune commented on Jul 10, 2012

benoitc commented on Jul 12, 2012

scott-w commented on Jul 13, 2012

athoune commented on Jul 13, 2012

tadeck commented on Jul 20, 2012

benoitc commented on Jul 23, 2012

pigmej commented on Jan 21, 2013

benoitc commented on Jan 21, 2013

pigmej commented on Jan 21, 2013

dirkmoors commented on Mar 14, 2013

RonnyPfannschmidt commented on Mar 14, 2013

wendall911 commented on Mar 14, 2013

pigmej commented on Mar 15, 2013

3 remaining items

pigmej commented on Mar 15, 2013

pigmej commented on Apr 10, 2013

dirkmoors commented on Apr 16, 2013

benoitc commented on Apr 16, 2013

pigmej commented on Apr 16, 2013

dirkmoors commented on Apr 16, 2013

benoitc commented on Apr 16, 2013

dirkmoors commented on Apr 16, 2013

pigmej commented on Apr 16, 2013

dirkmoors commented on Apr 18, 2013

silentjay commented on Jun 1, 2013

benoitc commented on Aug 30, 2013

isolationism commented on Jan 14, 2014

Huo12345 commented on Jan 15, 2014

benoitc commented on Aug 12, 2014

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions