[ home / board list / faq / random / create / bans / search / manage / irc ] [ ]

/hydrus/ - Hydrus Network

Bug reports, feature requests, and other discussion for the hydrus network.

Catalog

8chan Bitcoin address: 1NpQaXqmCBji6gfX8UgaQEmEstvVY7U32C
The next generation of Infinity is here (discussion) (contribute)
Name
Email
Subject
Comment *
File
* = required field[▶ Show post options & limits]
Confused? See the FAQ.
Embed
(replaces files and can be used instead)
Options
Password (For file and post deletion.)

Allowed file types:jpg, jpeg, gif, png, webm, mp4, swf, pdf
Max filesize is 8 MB.
Max image dimensions are 10000 x 10000.
You may upload 1 per post.

New user? Start here ---> http://hydrusnetwork.github.io/hydrus/
YouTube embed. Click thumbnail to play.

 No.849

win32

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v161/Hydrus.Network.161.-.Win32.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v161/Hydrus.Network.161.-.Win32.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v161/Hydrus.Network.161.-.OS.X.-.App.dmg

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v161/Hydrus.Network.161.-.OS.X.-.Extract.only.tar.gz

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v161/Hydrus.Network.161.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v161.tar.gz

I had a great week. I have increased video support for all platforms and fixed a lot of bugs.

animation

I have, for the most part, fixed animation on Linux and OS X. I could not get all gif transparency working, so some gifs with unusual palettes will render strangely, but otherwise, Linux and OS X should render animations and videos just like Windows. In my testing, they generally seem to work great, and I am very pleased. I have updated the static FFMPEG build that comes with the releases, and I added it for the Linux executable release (I think this was missing, before–it certainly wasn't working correctly, even if the user put it in themselves).

I also simplified my video rendering pipeline. It now uses less memory, has less latency, and, it feels to me, renders a little faster.

Furthermore, I have added mp4, wmv, and mkv support to my native rendering system. They now render just like webms, and in the same way do not have audio yet. I plan to add audio in the future.

I would like to know your experiences running these new video types in the client. If you experience crashes, memory errors, slowdown, or anything else, please let me know. There is room to improve my own system, and I would also like to add options for 'always launch mp4s in the default external program' and so on for those people who would prefer to default to MPC or VLC or whatever.

more processing improvements

I have switched the database to a slightly faster transaction mode that should speed things up a bit for all clients and possibly a lot for those with high latency hard drive connections, like laptops or encrypted drives. The drawback is that if your client crashes, there is a very small chance the database will forget the most recent writes to it. Let me know if you experience any significant changes in speed.

I also worked more on the processing code–it should accelerate/decelerate processing speed more quickly as needed and report rows/s more accurately.

BTW: In my testing these past weeks with rows/s, I have found that the easiest way to speed things up is just to keep my drive defragged. The database files can easily split into hundreds of pieces after a big operation, and leaving it like that just slows everything down. I use O&O Defrag and now have it automatically do a quick stealth defrag every twelve hours on all drives, which seems to keep things fairly smooth. It is another one of those things where five minutes a day causes less headache than an hour a month.

a/c dropdown and hover window fixes

I have fixed several bugs with the autocomplete dropdown–it should position and show/hide itself more accurately in Linux and OS X, and the bug that was stopping selecting system predicates with the enter key is fixed.

I cannot get the arrow keys (for selecting tag results) to fire on OS X because of an OS X-specific shortcut-binding behaviour I never knew about before. My long-term plan to improve shortcuts client-wide should eventually fix this, but for now, using ctrl+ or alt+arrow keys seems to work.

Also hover windows should position themselves a bit better for Linux and OS X. I still get some buggy behaviour in Linux, where the windows will sometimes not layout correctly or just decide to reposition in the top-left of the screen, but it happens less now. I will keep working on this.

full list

- updated windows python, sqlite and ffmpeg

- added linux ffmpeg binary to the executable build

- updated os x ffmpeg binary

- animation is generally working for linux and os x

- fixed some linux/os x ffmpeg calls

- fixed webm import for linux/os x

- webms work great

- moved to rendering gifs with PIL by default on linux and os x, which has bad palette support but renders more reliably

- the animation scanbar position caret is now visible on linux and os x

- fixed some timing/resume issues the animation scanbar on os x

- the old mediactrl video embed is removed

- mp4, wmv, mkv are added to native rendering control, no audio yet (just like webm)

- flv and flash will now have a little vertical padding on max zoom to make vertical hover windows easier to access

- changed database to a faster synchronisation mode

- tuned update processing daemon for more accurate time calculations and faster baseline speed, even if that knocks gui latency a bit

- fixed rows/s being thrown wildly off by long pauses

- made update processing daemon more reactive

- improved some of the flow and generally cleaned the update processing daemon

- added processing phase option to regular delay to processing of updates (this is useful if you run multiple clients and don't want them to process at the same time)

- fixed e621 tag parsing

- completely rewrote the focus engine behind the autocomplete dropdown

- fixed a bunch of autocomplete dropdown's generally buggy behaviour for linux and os x

- fixed autocomplete dropdown's display, sizing, and positioning for linux and os x

- fixed autocomplete dropdown selecting system predicates with the keyboard

- fixed autocomplete dropdown show status in linux with multiple pages open

- improved all subprocess calls, removing interim shell step and parameterising passed arguments

- improved how files and directories are launched in windows

- upnp is fixed for linux

- fixed hover window archive/inbox/delete icon buttons, which were spamming their commands to all open media viewers, not just their own

- added a simple raised border to hover windows to better delineate them from the canvas background

- updated the hover window size and position code to be a bit more reliable (still seems to bug out a bit on linux)

- tags hover window should now expand sideways more reliably

- it is now not possible to create nested export folders

- searching for numtags < x will now include files with no tags

- fixed manage boorus dialog OK in linux

- fixed booru selection mini-dialog in linux

- generally improved restoring database code

- fixed restoring a database when you have tag archives in existing database

- fixed a unicode encoding error when converting certain jobs to text (this was throwing errors in deviant art downloads/subscriptions–we'll see what was actually going on now, whether this is an error not being formatted right or something else)

- started some great rewrite of management panel gui code

- rearranged the download panel gui hierarchy slightly

- improved collapsible panel collapse/expand layout code

- removed code and database table for the old numerical ratings filter

- some general code refactoring and cleanup

- improved client upnp daemon timing

- harmonised how the client and server check requests for bandwidth-tracking eligibility

next week

I am on the cusp of committing myself to a severe and sorely needed overhaul of how the client's pages work behind the scenes, so I will start that first thing to maximise the chance that I will finish it in time for next Wednesday. I want to eventually have 100% accurate session support so that import pages will be resumable, and thumbnails will remember if they were selected, and so on, which means revamping some some integral stuff.

If I have time after that is done, I have some pixiv and danbooru parsing to look at and potentially adding native ugoira support.

I started reorganising my to-do list this week from a bunch of overloaded colour-coded Zhorn Software Stickies to a richer AbstractSpoon ToDoList Tasklist, and I am really pleased with the shift. I felt that some feature requests and bug reports were getting lost in the pile, so as I process my old lists of thoughts into proper actionable jobs, I hope I will be decreasingly able to lose track of them. If I have forgotten something you asked for a while ago and it is important to you, please tell me again and I will bump it up my priority list.

 No.850

File: 1434588707322.jpg (130.91 KB, 480x640, 3:4, 1293770894743.jpg)

I ran Visipics on my client files folder to try to find some duplicate files. It found about 500 files and I told it to move them to a separate folder. I was expecting to be able to just re-import them.

When I tried to import them it didn't recreate the files in the client files folder. Apparently it thought the files were already there and didn't bother to check at import time if the files were actually there, so it seemed to import but every file I tried to open said file not found.

Okay I can understand this. Here's where it gets weird.

I ran a quick integrity check and it says it deleted about 460 files, less than the folder full of files Visipics moved out.

I tried to re-import the files and I told it to delete files it successfully imported. It imported about 400 files successfully, but it didn't delete any files, and most disturbingly, about 100 files wouldn't import because it said the files were truncated.

Any idea why it's not deleting the files out of the dupe folder even though it's ostensibly importing them correctly? And why would 20% of the files fail to import now? I originally imported them back around Hydrus version 120, did the pillow library change since then to be more strict?

Memtest and chkdsk had no errors.


 No.851

>>850

BTW that image is for me, not hydrus_dev. My dumb ass playing with files in the client files folder.


 No.852

>>850

I would be interested in knowing a procedure to identify and remove duplicate images en masse, if anyone comes up with one that works.


 No.853

File: 1434604061969.jpg (403.61 KB, 1600x1050, 32:21, 0ada2a9e5a87f53db685661fac….jpg)

>>852

fdupes is a lovely little program

it can scan an entire foler and all subfolders, and wipe duplicates based on md5 hash

just expect to use command prompt, even on the windows fork

It's also insanely fast imo


 No.854

>>853

Hydrus inherently does this. Two images with the same hash are treated as "already in DB" when you import them.

Visipics doesn't care about hashes. It renders every image and finds pictures that look the same. It will find dupes where one is resized of the other, watermarked images, one PNG and the other JPG, etc.

It's the only free program I've ever found that can do this.


 No.855

File: 1434672915491.gif (1.99 MB, 308x241, 308:241, 03b3f7cc7e0266c85fab3c3739….gif)

>>850

Hydrus generally assumes that no-one touches its database folders. It keeps track of its files through a table in the database, not by monitoring the actual paths, so when you import a file, it calculates the file's hash and looks that up in the local files table. When you moved the files, the table entries were not affected, so as far as Hydrus knew, it still had the files. As you saw, it only threw an error if it tried to open the file in the media viewer, as this is pretty much the only time that file content is needed after an import.

I am not sure about the ~500/460 disparity. You can check for sure if the client is tracking more files than it actually has by running database->maintenance->delete orphan files (which will delete any files that Hydrus isn't tracking) and then comparing the number of local files in services->review services with the actual total number of files in client_files. If you distrust the local files count in review services, which is cached and may not have updated correctly with the recent weirdness, you can force it to recalculate via database->maintenance->clear service info cache. If the two numbers are not the same, then there is something wrong with my integrity checker, and I can look at it.

For a long time, I set files in the hydrus database to be read only so that someone wouldn't try to load one in photoshop and make a change, thus screwing up Hydrus's hashes. This turned out to be more of a pain than it was worth, so I recently stopped doing it. I can't remember if I un-read-only'd existing files, but I don't think I did. So, the files that were imported around v120 were probably still set to read only, meaning that when Visipics moved them they kept being read only, and so when Hydrus eventually tried to delete them after a successful import, it could not get permission to. This is all uncharted, though, so I am really not sure. You can check their read only status by looking at their right-click properties.

I do not know about the 100 'truncated' files. I think the importer tries to render the file at full res, dumping out if that fails, so if they were viewing fine inside the client before you moved them, I do not know why they would not subsequently render ok, unless Visipics changed them, but that seems unlikely. It may be that PIL/Pillow got more strict as you guess, and you just haven't looked at any of those hundred images in the past six months or whatever. I specifically moved to Pillow to be more accomodating, though, so I am not sure. If you like, email me one or more of the broken files, or post them here if they work, and I will look at them on my end.

>>852

I will eventually expand Hydrus's dupe detection to work like system:similar to en masse, but it is a big job.

Post last edited at

 No.856

>>855

You were correct about the files still being read only. I unset read only on them and it removed the files like normal.

As for the rest I'm trying to find a backup copy of one of them that wasn't the dupe Visipics found, so I can compare hashes and see if the file is somehow different. I'll post one if I can find the same hash and Hydrus won't import it.

So now I have a concern: if I do an integrity check and a file has been corrupted by bitrot, will Hydrus just throw it away? Can we move such files somewhere so we can manually repair them in case they are valuable and we're content to keep a repaired version?

I think Hydrus needs a general way to replace a file, e.g. export a file, import the new file, and copy all the old file's tags to the replacement, along with maybe a "replacement of:*hash*" tag.

As for the dupe detection, check out the hmsearch library.


 No.857

File: 1434744441334.jpg (315.22 KB, 822x2361, 274:787, bb65d2d86437ee06ddb29cd997….jpg)

>>856

Yeah, the thorough integrity check will delete. I wrote it to compensate for HDD failures, so I assumed files would be unrecoverable. I can set it to optionally move files to a user-chosen location in next week's release.

I want to write extensive 'file sibling' support once I have the better dupe searcher working. I want to add a new network service for describing the relationship of 'this file is a better version of this file, and they should both share the same tags', and then we can all collaborate on that problem.

That library looks really great, thank you for the recommendation.


 No.862

>>857

I would argue that you should make the move files option the default, because as far as I can tell the current integrity check doesn't log *which* files were lost, it just spits out "27 files deleted" or some such and you have no way to find out what happened.

I would say take the corrupt file and move it to some "corrupt" folder along with a Unicode text file that contains the tags the file had. It's not a good idea to use the [tags] export option here because the filename is limited to 255 characters on Windows.


 No.866

Hi there. I've been using Hydrus for about a week now and am starting to get the hang of it. Neat tool, so far. I have two questions though:

Subscriptions have an option to download tags if the image is in the db already. Is that available for regular image board downloads as well?

I have had a couple of failed image imports (server timeouts and tag errors when using a local tag archive). In the error message pane that appears in the lower right corner, is there a way to copy the image url? It would also be really handy to have a "Retry" button there, since a second attempt has so far fixed every timeout incident.

Thanks for your work!


 No.867

>>866

>Is that available for regular image board downloads as well?

If I understand your question correctly, the answer is that it depends on the board, because each image board has varying levels of support. Most boorus, for example, have a standard "creator" tag format that Hydrus can recognize and import as creator tags. But I think the same option is in there to import tags for existing files for everything. I often import the same files off my disk a couple times with different regex to pull all the relevant info out of their filenames.


 No.869

File: 1435081249988.jpg (4.12 MB, 2045x2754, 2045:2754, 27b5e56a4174a5bb504ec2afd1….jpg)

>>866

Both of these things will be available for all downloads pending a rewrite of the downloader engine that I am currently getting ready for. This week's work on management panels will be a big step towards it.

The 'download tags if file already in db' option is actually already there in any normal download page, but it is hidden because the download pipeline can't currently pass that variable to where it needs to be. The rewrite will fix this, and then I will unhide the checkbox.

For download failures, I want to add a listctrl that'll list all urls in all the download queues a page creates along with their status, something like this:

search 'delicious trap'

url_1 - successful

url_2 - successful

url_3 - failed (brief reason)

url_4 - successful

url_5 - downloading

url_6 - pending

And so on. You will be able to right-click on url_3 and select retry. I'll add a similar list for subscriptions so they can recover from failures better as well.

This second thing will be much more complicated than the first, but it is where I am heading towards. I also want download pages to remember their exact current status to make it so they can pause and resume through a client restart.


 No.871

>>869

Cool! I'll just kick back and wait fot that to happen then. Thanks for taking the time to reply.



[Return][Go to top][Catalog][Post a Reply]
Delete Post [ ]
[]
[ home / board list / faq / random / create / bans / search / manage / irc ] [ ]