Version 175

Name
Email
Subject
Comment *
File	Select/drop/paste files here
* = required field	[▶ Show post options & limits] Confused? See the FAQ.

Embed	(replaces files and can be used instead)
Options	Do not bump (you can also write sage in the email field) Spoiler images (this replaces the thumbnails of your images with question marks)
Password	(For file and post deletion.)
Allowed file types:jpg, jpeg, gif, png, webm, mp4, swf, pdf Max filesize is 8 MB. Max image dimensions are 10000 x 10000. You may upload 5 per post.

YouTube embed. Click thumbnail to play.

Version 175 hydrus_dev ## Board Owner 09/23/15 (Wed) 22:05:00 No.1160

win32

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v175/Hydrus.Network.175.-.Win32.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v175/Hydrus.Network.175.-.Win32.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v175/Hydrus.Network.175.-.OS.X.-.App.dmg

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v175/Hydrus.Network.175.-.OS.X.-.Extract.only.tar.gz

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v175/Hydrus.Network.175.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v175.tar.gz

I had a pretty good week, but I was fairly short on time. This release has several small improvements and fixes. I expect the next release to be in two weeks.

regex favourites

The 'regex shortcuts' button in the path tagging dialog has a slightly reformatted menu that now allows for saving your favourite regex phrases. They will copy to clipboard like the other entries, and you can also set a description for them.

custom animation start point

I had previously heard a joke that all youtube video could be safely skipped by ~20-30%, but apparently this has a semi-formal definition called the "Wadsworth Constant". So, under options->media, you can now specify where animations start by default. You can set whatever percentage you want.

When I move to animated thumbnails for gifs/video, I'll have them obey something similar to this, which should alleviate the common problem of having a page full of black thumbnails for video.

full list

- the 'regex shortcuts' button in the path tagging dialog now permits the managing of favourite regex phrases

- added 'start animations this % in' options to options->media

- animations will no longer clear to white while scanning beyond their immediate frame cache

- modifier-click (like shift-click) will now drag animations without altering their pause/play status

- the splash screen is a little larger, to better fit db update text in

- the splash screen's text is split into two lines, and shutdown repository processing is a bit more verbose

- fixed some incorrect dialog text in manage sibling and parent dialogs

- rewrote sibling and parent pair update code to deal with unusual situations more gracefully

- improved how the new downloader engine saves error notes

- the import review frame now lets you copy the selected entries' notes to the clipboard

- video files will no longer fail to import in windows because of unicode characters

- recycling files with unicode characters on Linux will work more reliably, and when it breaks, it will fail more gracefully

- the client and server can now detect and recover from a rare error where a database file is created but never initialised with the appropriate tables

- the tag/file upload popup message now dismisses itself after a few seconds, rather than pointlessly lingering

- wx event ids used in one-shot menus will be recycled, which should alleviate the 'after a long time, some menu selections stop doing anything' problem

- some code cleanup

next week

I am now going to concentrate on converting gallery downloads and subscriptions to the new downloader engine, where they will generally work better, allow for download reattempts, be easier for me to customise and extend, and survive through client restarts. This has been a long time coming, and it doesn't look like it will be simple or quick. I expect it will take me two weeks, although it may take more or less. I will make a post next Tuesday with an update, and then hopefully a 'Release Tomorrow!' post the week after.

Please keep sending your bug reports and thoughts in as normal, but unless something is an emergency, please expect a short delay before I can work on anything else.

Anonymous 09/24/15 (Thu) 04:35:52 No.1161

File: 1443069366106.gif (184.98 KB, 200x310, 20:31, 1375643293110.gif)

Woo!

Anonymous 09/24/15 (Thu) 21:56:27 No.1162

File: 1443131787488.jpg (63.48 KB, 960x720, 4:3, feff475814675f8f806a6c45c9….jpg)

Very often, when I click and drag a thumbnail onto 8chan's file upload box, I will be unable to click and drag any more thumbnails anywhere until I restart the client.

It has happened just now. I can successfully add this first image, but any after just won't drag. That's not to say that I'm selecting multiple images and dragging them all at once; I am trying to add them individually, one at a time.

In the past, I could just keep trying and it would eventually work, but lately that hasn't been the case.

v175, win7.

Let me know if any other information would help.

Thanks dev, hydrus is great. Never give up! Work hard and do your best.

Anonymous 09/25/15 (Fri) 02:01:56 No.1163

>>1162

I've had this issue, too. I don't have to restart. Instead, I try dragging other images until I find one that works, and eventually it'll work.

hydrus_dev ## Board Owner 09/26/15 (Sat) 19:48:25 No.1164

File: 1443296906301.gif (10.86 KB, 492x159, 164:53, c26b94ab834aff3bcc8b74cd8a….gif)

>>1162

>>1163

I have read that this sort of thing can happen when Windows gets confused about mouse drivers or program privileges (e.g. you are running the client as admin, but not your browser), and it forgets or is somehow otherwise unable to clean up the object that represents the first drag and drop, so the originating program never gets that message and can't tidy itself up properly and hence start a new drag and drop. Apparently, a simple fix is either to just try hitting escape when the originating window has focus, which can sometimes force the reset, or try dragging something new onto the originating window–you don't have to drop it, as I understand, just dragging any old file or bit of selected text temporarily over the originating window will cause it to reset its drag and drop status.

Please let me know if either of those works.

Anonymous 09/30/15 (Wed) 12:27:33 No.1169

Just to be curious, should the importing be taking such a long time? It's taking me around 3-4 seconds per image, in a defragged disk with nothing else running. I know SHA isn't the fastest hashing method, but I was curious if I was doing something wrong, or if there's a way to accelerate things or something. Your program is looking too good to abandon it now, but this is getting unbearable.

hydrus_dev ## Board Owner 09/30/15 (Wed) 15:54:22 No.1170

File: 1443628462704.jpg (386.28 KB, 1782x1335, 594:445, b3ee0a2d652fbf3a8628c43e9b….jpg)

>>1169

A normal hdd import should process as fast as your computer allows. If you have particularly complicated files, like 4000x4000 jpegs or 8MB gifs or whatever, there will usually be a bit of a delay, but otherwise they should stream in typically every 0.2-1.0 seconds. Compared to other bottlenecks like hdd latency, SHA takes no time at all.

Try going help->debug->db profile mode while you are importing some stuff. That'll dump a lot of rich timing information to your log, which is stored in install_dir/logs/client.log. Scroll down to the bottom of that file and find some tables of numbers for the import_file action, and then email that text to me, or dump it here if it'll fit, or pastebin it, or just email me your whole log file if you like. I'll have a look at what is taking so long.

Anonymous 10/01/15 (Thu) 17:46:28 No.1171

So I defragged again (for some reason it went to 30% fragmentation since I started adding files from a mere 3%). So after maybe 20 hours of defragging it improved greatly, at worst 1 per second but now it can crunch 2-3 in 1s. Maybe my computer got a little bit upset with all the extra files. All is better now, thanks for the help!

Just to be curious, though. Why do you create two thumbnails? I was checking my defrag and there were tons of thumbnails fragmented.

Anonymous 10/02/15 (Fri) 12:22:57 No.1173

I still think you should see some of the logs, these files were all <900x900 of around 25kb each, took around a second.

When it's importing, the menu lags a bit (fluid for a second, frozen in the next) but I guess it's natural. Maybe that's what's making everything take longer than it should.

Thanks for your assistance.

http://pastebin.com/qhuyj5f8

hydrus_dev ## Board Owner 10/02/15 (Fri) 16:33:56 No.1175

File: 1443803636614.jpg (103.46 KB, 1280x861, 1280:861, 5efb82ba6c50042ef467b61129….jpg)

>>1171

>>1173

Thank you for this update. Those db profiles all look great. Your machine is loading those files in ~50-300ms, which suggests hard drive is not slowing things down too much. There might be some unexpected gui lag for some reason–maybe the resized thumbnail generation, which I recently moved to happen outside of the db transaction. There might be some hard drive lag I am not capturing. Do your imports' thumbnails fade in cleanly or with a permanent stutter?

Perhaps I should write a gui profile mode.

I am not sure what to do about fragged hard drives. I don't know if the particular way I copy files around is contributing to the problem, or if Windows/OSes in general are shit at allocating contiguous blocks when you throw a lot of small files at them. I don't know of any easy way I can say hey_windows_defrag_this_file( path ), so it may just have to be up to the user to keep things defragged. I used to use O&O defrag, and now I use PerfectDisk–if you are looking for a good permanent defrag solution, I can recommend both.

Most gui stuttering during high db activity like a hard drive import is usually when the db completes a job (writing it to disk), which impolitely eats cpu until it is done, making everything else in the program has to wait for it. When this is just adding a couple of tags it takes a few milliseconds, but big stuff can put the gui on pause for half a second. There isn't really a fix for this, so I've been moving high db stuff away from the user's notice, like the recent move of update processing to idle time.

For thumbnails, I create one thumbnail at 200x200 when the file is imported, and then the first time the gui wants to load a thumb, lets say at 120x80, it generates that resized thumb from the 200x200, rather than the original file. Doing 200x20->120x80 is much faster than 4000x3000->120x80, so it makes new results load a lot faster, and allows the user to change their thumbnail size without incurring a huge lag cost of having to regenerate from scratch.

200x200 thumbs are also how the file repositories store and distribute thumbs, so everything all fits neatly together across the network, no matter what size thumbnail the client prefers on their end.

I've rambled a bit here. In any case, I will keep my eye on ways to speed things up, and I will experiment with larger transfer block sizes to see if that reduces fragging. Please let me know if this problem returns or if you discover anything new!

Anonymous 10/02/15 (Fri) 17:16:43 No.1176

>>1175

They fade in cleanly. When they do, I can move around the mouse over the buttons and they show the highlight, just for a bit, then it freezes again. Whenever the "db locked" message pops up, it "unfreezes" for a bit and then it gets stuck again until the message reappears.

About the fragmentation of files, I guess it's part of the Windows package, can't do nothing about it. I usually defrag without booting into windows from the commandline, for a faster, tidier and more thorough result.

Another note, whenever a "failed" or "already in db" import occurs, it takes more or less the same time to go to the next file, is that useful?

I just finished typing that and the client threw an error:

The client is running out of memory (according to windows 1098mb of RAM), unknown caller probably GUI

I'm at the import tab, and it has around 38k files loaded (my faulty maybe?).

It also spat this:

Database Error!

Unknown Caller, probably GUI.

Traceback (most recent call last):

File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.HydrusDB", line 196, in _ProcessJob

File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.ClientDB", line 6442, in _Write

File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.ClientDB", line 5068, in _SetJSONDump

File "C:\code\Hydrus\build\client\out00-PYZ.pyz\json", line 243, in dumps

File "C:\code\Hydrus\build\client\out00-PYZ.pyz\json.encoder", line 210, in encode

MemoryError

Until next time.

Anonymous 10/02/15 (Fri) 17:22:01 No.1177

This was in the client.log after I shut it down.

The following exception occured at 2015/10/02 14:18:02:

DBException: (u'', 'Unknown Caller, probably GUI.', 'Traceback (most recent call last):\n\r\n File "C:\\code\\Hydrus\\build\\client\\out00-PYZ.pyz\\include.HydrusDB", line 196, in _ProcessJob\n\r\n File "C:\\code\\Hydrus\\build\\client\\out00-PYZ.pyz\\include.ClientDB", line 6442, in _Write\n\r\n File "C:\\code\\Hydrus\\build\\client\\out00-PYZ.pyz\\include.ClientDB", line 5068, in _SetJSONDump\n\r\n File "C:\\code\\Hydrus\\build\\client\\out00-PYZ.pyz\\json", line 243, in dumps\n\r\n File "C:\\code\\Hydrus\\build\\client\\out00-PYZ.pyz\\json.encoder", line 210, in encode\n\r\nMemoryError\n')

File "C:\code\Hydrus\build\client\out00-PYZ.pyz\threading", line 783, in __bootstrap

File "C:\code\Hydrus\build\client\out00-PYZ.pyz\threading", line 810, in __bootstrap_inner

File "C:\code\Hydrus\build\client\out00-PYZ.pyz\threading", line 763, in run

File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.HydrusDB", line 275, in MainLoop

File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.HydrusDB", line 208, in _ProcessJob

File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.ClientDB", line 4206, in _ManageDBError

File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.HydrusData", line 771, in ShowExceptionDefault

waiting for daemons to exit

running maintenance

fattening service info

Exception in thread Thread-2:

Traceback (most recent call last):

File "C:\code\Hydrus\build\client\out00-PYZ.pyz\threading", line 810, in __bootstrap_inner

File "C:\code\Hydrus\build\client\out00-PYZ.pyz\threading", line 763, in run

File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.ClientImporting", line 158, in _THREADWork

File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.ClientController", line 487, in PageDeleted

File "C:\code\Hydrus\build\client\out00-PYZ.pyz\wx._core", line 16709, in getattr

PyDeadObjectError: The C++ part of the FrameGUI object has been deleted, attribute access no longer allowed.

exiting client

exiting db

The following exception occured at 2015/10/02 14:18:07:

ShutdownException: Application shutting down!

File "C:\code\Hydrus\build\client\out00-PYZ.pyz\threading", line 783, in __bootstrap

File "C:\code\Hydrus\build\client\out00-PYZ.pyz\threading", line 810, in __bootstrap_inner

File "C:\code\Hydrus\build\client\out00-PYZ.pyz\threading", line 763, in run

File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.ClientImporting", line 174, in _THREADWork

File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.HydrusData", line 771, in ShowExceptionDefault

hydrus client shut down at Fri Oct 02 14:18:08 2015

hydrus_dev ## Board Owner 10/10/15 (Sat) 17:12:30 No.1205

File: 1444497150964.jpg (1023.98 KB, 2304x4096, 9:16, 1875a6e2dc65bd4baf0a9eb11f….jpg)

>>1176

>>1177

Thank you for this. It is interesting that is still delays on a deleted/redundant file. It seems something outside the db is slowing things down. I will keep looking at it, and probably add a gui profile mode.

That memory error is odd. Someone else says they have memory problems with >10,000 files in a view. With 38k taking up ~1GB, that means each thumb is eating about 27KB of data. I don't mean that to be true, so perhaps thumbnail bmps are being generated in the background or not getting cleaned up like I want. I'll look into it.

However JSON also sometimes gives MemoryErrors when it comes across malformed data, so let me know if you get that again when there is no memory bottleneck.