Version 184

Name
Email
Subject
Comment *
File	Select/drop/paste files here
* = required field	[▶ Show post options & limits] Confused? See the FAQ.

Embed	(replaces files and can be used instead)
Options	Do not bump (you can also write sage in the email field) Spoiler images (this replaces the thumbnails of your images with question marks)
Password	(For file and post deletion.)
Allowed file types:jpg, jpeg, gif, png, webm, mp4, swf, pdf Max filesize is 8 MB. Max image dimensions are 10000 x 10000. You may upload 5 per post.

YouTube embed. Click thumbnail to play.

Version 184 hydrus_dev ## Board Owner 12/03/15 (Thu) 00:58:32 No.1527

win32

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v184/Hydrus.Network.184.-.Win32.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v184/Hydrus.Network.184.-.Win32.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v184/Hydrus.Network.184.-.OS.X.-.App.dmg

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v184/Hydrus.Network.184.-.OS.X.-.Extract.only.tar.gz

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v184/Hydrus.Network.184.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v184.tar.gz

Despite being ill the whole time, I had a great week. I have completely cleared out my old bug to-do list and I added support for external file storage.

file storage location(s)

In the new options->file storage locations panel, you can now designate additional folders for your client to use to store files. You give each location a weight, and the client will incrementally balance all your files proportionally across them. Once you have a desired balance set, the client will move small amounts at a time in the background, or you can force an immediate full rebalance from database->maintenance->rebalance file storage. I hurried this system to solve two problems:

1. Your collection is larger than any single hard drive you own.

2. You want to run client.db from a small fast drive like an SSD but would prefer to keep your read-only files on a larger slower HDD.

This is still in a prototype stage, so please do not play with it if you are a new user or do not maintain a backup.

I have attempted to make this system flexible and to fail gracefully. If you click on a thumbnail and the file or location responsible for that file is missing (e.g. you unplugged the usb hard drive they were on), you will get some popup errors but nothing terrible will happen. If the system discovers subfolders in the wrong location (e.g. your drive paths changed and you are trying to put everything back together), it will silently attempt to heal the damage by putting things where they are supposed to go. You can remove the original install_dir/db/client_files from the list if you like (as long as you've added something else to take over), and that folder will slowly lose responsiblity for storing anything at all.

If you use this, let me know how you get on! I'm obviously interested in any bugs you encounter. I expect I will write some formal help for it, so let me know whatever confused you. Some common-sense rules apply for this initial release–don't try to share the same external storage location between different clients, and don't force a rebalance while watching your webms, as the system won't be able to move open files.

maintenance and processing

If you have special maintenance and processing idle-timing options set, please recheck them, as that panel has had a minor makeover! Figuring out when idle time will kick in should be a bit more intuitive now!

autocomplete improvements

I have tweaked the low-latency optimisations I added to the autocomplete entry last week, and also added a couple of shortcuts:

Tab – This will now force the autocomplete to go fetch autocomplete results immediately. A new option in options->speed and memory also lets you completely disable automatic result fetching, so use Tab if you want to only fetch results manually.

Shift+Tab – This will disable/enable capture of Up/Down/Enter key events so you can enter non-english text with an IME without having your IME key events stolen for the dropdown list.

waiting politely indicator

Whenever a downloader waits for a few seconds, instead of spamming 'waiting politely' status text, it will now turn a little 'traffic light' icon from green to yellow. If you do a lot of downloading, let me know what you think of this! I expect I could do more with it. Turning red on an error, perhaps.

full list

- added external client_files storage!

- you can add external client_files folders in options->file storage locations, further giving them weight

- a new daemon will incrementally rebalance your files (and recover orphaned subfolders!) over your different storage locations

- you can also force a full rebalance from the new database->maintenance->rebalance file storage

- simplified the maintenance and processing panel

- the maintenance and processing panel controls now appropriately support being set to 'none'

- the 'run jobs on idle?' question is now explicit on the maintenance and processing panel

- deselecting 'run jobs on idle/shutdown' will now disable subordinate controls

- hitting tab on the autocomplete control now triggers an immediate 'fetch results' call

- added a checkbox to options->speed and memory to completely disable automatic autocomplete results fetching (i.e. if you want to manually control a/c result-fetching only with this new tab shortcut)

- the new less-laggy autocomplete results fetch won't trigger if the latest query is shorter than the cached query (i.e. you won't get lag when hitting backspace a bunch of times on autocomplete)

- hitting shift+tab on the autocomplete control now disables all other key event capture, letting you enter IME without your up/down/enter presses controlling the dropdown list of results

- 'waiting politely' time is now indicated with a small 'traffic light'-type circle control on all downloader pages

- fixed some occasional 'I'll go sit in the top-left corner of the screen and not fix my position' hover window behaviour in Linux

- fixed sibling predicate collapse for the 'read' autocomplete dropdown for database results

- refactored how predicates are collapsed and sorted to be a bit more sensible

- fixed maintenance of local booru data use even if local booru does not receive any requests

- improved some service/content update error handling in repository sync

- the system:rating value-entry dialog no longer lists its rating services in random order

- 'open externally' will no longer show for non-local thumbnails' right-click menus

- fixed audio/pdf thumbnail display for local booru, although they will be full size for now

- harmonised some hex-prefix folder generation code

- improved some file and directory copying code

- improved upload pending popup message cleanup

- added help_dir to hydrusconstants

- miscellaneous refactoring

- misc cleanup

- extracted server-specific services and server resources from the common import path to the server import path

- a bit more misc server refactoring

- updated a couple of bits of help

next week

My local to-do list is now free of bugs, and I don't think I have anything new to add that is too urgent, so I think I can focus more on my github queue. So, I'll work on that, and do some code cleaning and add some more unit tests.

Anonymous 12/03/15 (Thu) 06:29:44 No.1530

Awesome job

Anonymous 12/03/15 (Thu) 09:49:49 No.1531

File: 1449136189637.gif (663.81 KB, 500x500, 1:1, awwyeeah.gif)

Thank you for your hard work. Hail hydrus.

Anonymous 12/04/15 (Fri) 06:41:10 No.1534

prefer the the text prompts over the traffic lights for the waiting politely indicator

i just find the graphic not fitting the overall aesthetic

not really a big deal though, thanks for the work as always

hydrus_dev ## Board Owner 12/04/15 (Fri) 17:08:04 No.1536

File: 1449248884990.jpg (1.68 MB, 1062x1610, 531:805, 6f8fed98dda32b51186df3296e….jpg)

>>1534

Thank you for the feedback! I'll add an option to switch the traffic light out for just a bit of text in the same location.

Anonymous 12/05/15 (Sat) 19:01:45 No.1543

What is the weight based on? Size or count or something else?

Anonymous 12/05/15 (Sat) 19:03:25 No.1544

What happens if a storage location goes down during an integrity check? Didn't the database remove entries for missing files before?

Anonymous 12/05/15 (Sat) 19:15:02 No.1545

My most desired new features would be:

update Pixiv scraper for manga, ugoira and comic pages

mass duplicate search, maybe a way to merge tags of duplicates

one-click add a saved regex and namespace to import tag dialog

limit search results to ~5K so huge searches don't crash

Anonymous 12/06/15 (Sun) 14:41:05 No.1546

>>1545

I second the desire for an updated Pixiv scraper!

hydrus_dev ## Board Owner 12/06/15 (Sun) 18:47:42 No.1547

File: 1449427662583.jpg (395.14 KB, 1608x2001, 536:667, 0e5641011545954fc2d553162a….jpg)

>>1543

The new rebalancing daemon spreads the client_files subfolders across the locations randomly proportionally to their assigned weight, like a ratio. There are 256 folders, 00 to ff, and since files are allocated to each folder essentially at random (the folder name is the first two characters of the sha256 hash), they tend to be about the same size. If you have two locations with weight 1, each will get 128 subfolders. If you have three folders with weight 2, 2, 5, they will get approx 57, 57, 142.

If you have two mostly empty drives set aside for your client's storage, say 200GB and 300GB, you could give their respective locations a weight of 2 and 3, and client_files will be spread roughly proportionally across them, meaning as your collection grows, they will fill up at about the same percentage rate.

If this system proves to be a problem, I can make it more complicated, but for now, this is a really quick and easy way to keep track of the paths. I might add 'don't let this folder get bigger than x GB', if people want it, but I am not sure how much people actually would use it. If you are interested in the current system, have a play with it and let me know what you think. Perhaps I should reword the option panel to better explain how the weight works.

>>1544

Yeah, I think files' db rows would be deleted, so don't run an integrity check if your usb drive is unplugged!

Under normal operation, however, the client never usually checks that its files actually exist. It is usually only when you actually click on a thumbnail that the real file needs to be loaded (to generate an image for the preview canvas). If your usb drive is unplugged when that happens, the client will throw up some FileNotFound error popups, but it won't remove those files from its records.

>>1545

>>1546

My current plan is to clear out my backlog of small fixes and cleanups, and then to make a big poll with about twenty 'new things' on it so people can vote and so I can see what people want in a more quantitative way. Pixiv and mass dupes will definitely be on the list, but I'll try to remember to canvas for the poll the week before anyway. Please remind me when that time seems to be happening.

I am not sure what the one-click saved regex thing is, so can you explain that more?

The limiting search result option should be pretty easy–I think we have talked about it somewhere else?–so I will try to slip it in this week.

Anonymous 12/08/15 (Tue) 01:01:24 No.1552

>Tab – This will now force the autocomplete to go fetch autocomplete results immediately.

This really helped me out. Thanks

Anonymous 12/12/15 (Sat) 03:59:53 No.1566

>>1547

Saving regex is good, but being able to save both the regex and a namespace would be better, then being able to add them both quickly to the import box. Pasting the regex then typing the namespace gets tedious.

Anonymous 12/13/15 (Sun) 14:12:08 No.1569

File: 1450015928840.gif (497.94 KB, 270x169, 270:169, 8d79088b89c0c54bbedd6b125b….gif)

>>1547

So, how can i use the reblancing daemon to completely move all files to a new location?

hydrus_dev ## Board Owner 12/13/15 (Sun) 17:38:34 No.1570

File: 1450028314764.jpg (749.89 KB, 1280x1733, 1280:1733, 3cda74b073453770378b63b0c8….jpg)

>>1566

I see, thanks. That's a great idea. I will make a note to add another type of custom entry that also has namespace, and instead of copying to clipboard, selecting the menu item will prefill both fields.

>>1569

Go to file->options->file storage locations and add your desired (preferably empty!) destination folder. Then delete your install_dir\db\client_files folder from the list and click save. The daemon will slowly move all your files from your old client_files to your new place over the next few days, or you can force it to go all at once from database->maintenance->rebalance file storage.

Let me know if it doesn't work for you.