uploading bug

Name
Email
Subject
Comment *
File	Select/drop/paste files here
* = required field	[▶ Show post options & limits] Confused? See the FAQ.

Embed	(replaces files and can be used instead)
Options	Do not bump (you can also write sage in the email field) Spoiler images (this replaces the thumbnails of your images with question marks)
Password	(For file and post deletion.)
Allowed file types:jpg, jpeg, gif, png, webm, mp4, swf, pdf Max filesize is 8 MB. Max image dimensions are 10000 x 10000. You may upload 5 per post.

File: 1438564073735.jpg (15.36 KB, 210x210, 1:1, a044ad345982b2b7f0e3421430….jpg)

uploading bug Anonymous 08/03/15 (Mon) 01:07:53 No.987

Hey hydrus dev, got a bug, not sure whats wrong

have someone else hosting the repo itself, but desu image repo is up, people are able to download from it, test it in manage services and it gives an "everything is working"

but, when i try to upload images to the repository, it times out on me, i have 169 images waiting to be uploaded but every time i hit upload, it just times out on me, any ideas?

hydrus_dev ## Board Owner 08/04/15 (Tue) 17:56:10 No.993

File: 1438710970486.jpg (379.99 KB, 1339x1719, 1339:1719, b477c8311dac012083bb28988e….jpg)

I am not sure. Uploading and downloading is working fine on my test system, so I think that code is ok.

I suspect that your repo has locked its db somehow, so it can't complete complicated http requests any more. If the guy running the server restarts it (which will probably need a force-close) and checks server.log, there will probably be something interesting in there–some maintenance routine that got caught, or hard drive access error, or something else unexpected. Neither the account check (the 'everything is working' bit) nor file reads require the db, so that is probably it. Are clients still synching with it ok? Update generation should require db access (maybe it is this that is stuck in a loop!).

If there are interesting tracebacks, you can either post them here or email me the whole log.

Anonymous 08/05/15 (Wed) 17:13:27 No.999

The server logs are rather terse. They only contain startup/shutdown messages and a very old pair of errors:

The client could not delete the temporary file /tmp/hydrustEbZFt
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/twisted/internet/base.py", line 1203, in mainLoop
    self.runUntilCurrent()
  File "/usr/lib/python2.7/site-packages/twisted/internet/base.py", line 798, in runUntilCurrent
    f(*a, **kw)
  File "/usr/lib/python2.7/site-packages/twisted/internet/defer.py", line 393, in callback
    self._startRunCallbacks(result)
  File "/usr/lib/python2.7/site-packages/twisted/internet/defer.py", line 501, in _startRunCallbacks
    self._runCallbacks()
--- <exception caught here> ---
  File "/usr/lib/python2.7/site-packages/twisted/internet/defer.py", line 588, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/opt/hydrus/include/HydrusServerResources.py", line 307, in _callbackRenderResponseContext
    if do_finish: request.finish()
  File "/usr/lib/python2.7/site-packages/twisted/web/server.py", line 228, in finish
    return http.Request.finish(self)
  File "/usr/lib/python2.7/site-packages/twisted/web/http.py", line 926, in finish
    "Request.finish called on a request after its connection was lost; "
exceptions.RuntimeError: Request.finish called on a request after its connection was lost; use Request.notifyFinish to keep track of this.

/usr/lib/python2.7/site-packages/PIL/Image.py:2221: DecompressionBombWarning: Image size (90510000 pixels) exceeds limit of 89478485 pixels, could be decompression bomb DOS attack.
  DecompressionBombWarning)
ERROR: invalid element ID with leading byte 0x0
ERROR: invalid element ID with leading byte 0x0

This is the server.log, but it says "the client could not delete…." – I suspect that's an oversight in error handling. As for the decompression bomb, who knows but it seems legit.

I restarted it and opened it in a terminal, and this is part of what it printed (not logged):

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/twisted/internet/base.py", line 1203, in mainLoop
    self.runUntilCurrent()
  File "/usr/lib/python2.7/site-packages/twisted/internet/base.py", line 825, in runUntilCurrent
    call.func(*call.args, **call.kw)
  File "/usr/lib/python2.7/site-packages/twisted/internet/defer.py", line 393, in callback
    self._startRunCallbacks(result)
  File "/usr/lib/python2.7/site-packages/twisted/internet/defer.py", line 501, in _startRunCallbacks
    self._runCallbacks()
--- <exception caught here> ---
  File "/usr/lib/python2.7/site-packages/twisted/internet/defer.py", line 588, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/opt/hydrus/include/HydrusServerResources.py", line 414, in _errbackHandleEmergencyError
    self._CleanUpTempFile( request )
  File "/opt/hydrus/include/HydrusServerResources.py", line 494, in _CleanUpTempFile
    HydrusFileHandling.CleanUpTempPath( os_file_handle, temp_path )
  File "/opt/hydrus/include/HydrusFileHandling.py", line 53, in CleanUpTempPath
    os.close( os_file_handle )
exceptions.OSError: [Errno 9] Bad file descriptor

I suspect that might have something to do with it. Also you should probably be using file objects rather than file descriptors, I suspect they will have more graceful failure handling

Ironically, that error seems to have stopped Hydrus from reaching:

        print( failure.getTraceback() )

in _errbackHandleEmergencyError. So I suspect the real error is being masked by this weird file descriptor issue.

For now I'm going to patch that function to read like this:

    def _errbackHandleEmergencyError( self, failure, request ):
        
        try:
            self._CleanUpTempFile( request )
        finally:
            print( failure.getTraceback() )
            
            try: request.write( failure.getTraceback() )
            except: pass
            
            try: request.finish()
            except: pass

Not perfect but maybe it'll give me a better idea of what's going on. I'll update you if this comes up with anything interesting.

Strangely, I seem to be able to upload from my local machine (the one I'm hosting on) and I am 100% sure that a successful connection is being made to the hydrus server when >>987 is trying to upload to it.

I have a reasonable knowledge of Python, so if you have any more functions or areas you'd like to see some debug prints in, tell me and I'll try them out.

Anonymous 08/05/15 (Wed) 17:37:13 No.1000

Okay you'll never guess what error it printed

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/twisted/internet/base.py", line 1203, in mainLoop
    self.runUntilCurrent()
  File "/usr/lib/python2.7/site-packages/twisted/internet/base.py", line 825, in runUntilCurrent
    call.func(*call.args, **call.kw)
  File "/usr/lib/python2.7/site-packages/twisted/internet/defer.py", line 393, in callback
    self._startRunCallbacks(result)
  File "/usr/lib/python2.7/site-packages/twisted/internet/defer.py", line 501, in _startRunCallbacks
    self._runCallbacks()
--- <exception caught here> ---
  File "/usr/lib/python2.7/site-packages/twisted/internet/defer.py", line 588, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/opt/hydrus/include/HydrusServerResources.py", line 258, in _callbackRenderResponseContext
    self._CleanUpTempFile( request )
  File "/opt/hydrus/include/HydrusServerResources.py", line 495, in _CleanUpTempFile
    HydrusFileHandling.CleanUpTempPath( os_file_handle, temp_path )
  File "/opt/hydrus/include/HydrusFileHandling.py", line 53, in CleanUpTempPath
    os.close( os_file_handle )
exceptions.OSError: [Errno 9] Bad file descriptor

I thought my patch didn't work until I realized it was printing this with a different stack trace. Argh.

Reverting my previous patch and making CleanUpTempPath just print_exc if it fails to close gave me this:

hydrus server started at Wed Aug  5 12:27:41 2015
Traceback (most recent call last):
  File "/opt/hydrus/include/HydrusFileHandling.py", line 54, in CleanUpTempPath
    os.close( os_file_handle )
OSError: [Errno 9] Bad file descriptor
The client could not delete the temporary file /tmp/hydrusmGk6P0

However that's even less useful. Somehow the stack here is empty too. I think CleanUpTempPath might be being called twice, but that doesn't explain the timeouts in the first place.

Anonymous 08/08/15 (Sat) 18:23:28 No.1005

>>1000

Did you install from source, if so is there a symlink or binary of ffmpeg in the bin folder?

This should only matter if a webm is attempting to be uploaded I got a similar error when installing from source. (the file descriptor not decompression bomb)

hydrus_dev ## Board Owner 08/11/15 (Tue) 20:44:04 No.1013

File: 1439325845064.jpg (351.67 KB, 1272x1540, 318:385, 2b9645572ddc1925296e3d3b9d….jpg)

Thank you for this great reporting. I am sorry I didn't have more time this week to work on it.

I think there might be an error after the temp file cleanup in callbackRenderReponseContext, so it jumps to the errback, which then tries to temp file cleanup again. Or maybe some library is closing the file descriptor on certain kinds of error, I am not sure.

Anyway, it looks like I wrongly assumed my temp file cleanup would work 100% reliably, so I put it in dangerous places without thinking. For tomorrow, I've added another try block around the os.close call, so if that fails, it won't dump out, and I've removed the temp_file_info variable so it won't try to run twice anyway. We'll see what errors it produces in the new version.

When I moved to the tempfile library, I tried using their higher level stuff to get a file handle, but then I had to pass a path to some of my media parsing libraries, and I had trouble with them trying to open/close the path while tempfile had its handle on it. I got mkstemp to work, although it is ugly, so I left it there. Do you know of a better solution for this sort of thing?

Anonymous 08/12/15 (Wed) 08:18:35 No.1020

>>1005

I did install from source. There's a ffmpeg binary but I didn't include the upnpc_linux binary (hopefully not related to this issue).

>>1013

If you use tempfile.NamedTemporaryFile, that has a "name" attribute containing the path of the file. The docs do have a caveat here saying it may not be able to be opened by other programs on Windows, but I don't have a Windows machine to confirm whether or not this applies to opening files for reading.

Relating to the original issue though, do you know what function names I should be looking out for when it comes to receiving files and processing received files on the server side?

hydrus_dev ## Board Owner 08/12/15 (Wed) 18:25:15 No.1023

File: 1439403915289.jpg (107.99 KB, 512x512, 1:1, 978854813dd8cbb4a447a44c38….jpg)

>>1020

On the serverside:

_callbackParsePOSTArgs in HydrusServerResources will generate this temp path that is causing trouble and write the file from the http POST request to it.

ParseFileArguments in HydrusServerResources will then check the file and prep it for sending to the db. It will generate the thumbnail here. I think this is where that decompressionbomb warning would be generated, in the HydrusFileHandling.GetFileInfo call, which would end up at HydrusImageHandling.GetResolutionAndNumFrames, which attempts to load it in PIL. Maybe when this dumps out, it doesn't release the file handle on the temp file? Maybe not quick enough? The garbage collection in the cleanup function was an attempt to deal with this, but I might have put it in the wrong place.

_AddFile in ServerDB will receive the file and try to import it. File and thumbnail are written to your db folder here.

Assuming no exceptions were thrown, a content-less 200 OK should be written to the http request in _callbackRenderResponseContext, the first line of which attempts to clean up the temp path if the request has one.

_callbackRenderResponseContext is generally expected to work. If it throws an exception because of an odd error like a client ditching the connection or some system problem, _errbackHandleEmergencyError takes over by just dumping exception text to the request and closing it. Unfortunately, the emergency handler, which really is expected to work 100% of the time, was getting an exception when trying to close the temp_path, so it would never tell the request to finish, and so your response hung.

Anonymous 08/28/15 (Fri) 06:05:39 No.1082

>>1023

so, how does one fix it?

hydrus_dev ## Board Owner 09/01/15 (Tue) 18:19:41 No.1089

File: 1441131581671.jpg (241.45 KB, 918x1392, 153:232, 44a2dc45009d87a386ebc7ab69….jpg)

>>1082

Do you have any new error messages in your server.log? The tempfile stuff should work better, and I have made some error handling changes the past few weeks, I think including to the server, so make sure you are running the newest version, and then try again.

The server is going headless in tomorrow's release, by the way. It shouldn't change much for this discussion, except it now obviously runs from a command line.