Version 176

Name
Email
Subject
Comment *
File	Select/drop/paste files here
* = required field	[▶ Show post options & limits] Confused? See the FAQ.

Embed	(replaces files and can be used instead)
Options	Do not bump (you can also write sage in the email field) Spoiler images (this replaces the thumbnails of your images with question marks)
Password	(For file and post deletion.)
Allowed file types:jpg, jpeg, gif, png, webm, mp4, swf, pdf Max filesize is 8 MB. Max image dimensions are 10000 x 10000. You may upload 5 per post.

YouTube embed. Click thumbnail to play.

Version 176 hydrus_dev ## Board Owner 10/07/15 (Wed) 23:01:13 No.1190

win32

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v176/Hydrus.Network.176.-.Win32.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v176/Hydrus.Network.176.-.Win32.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v176/Hydrus.Network.176.-.OS.X.-.App.dmg

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v176/Hydrus.Network.176.-.OS.X.-.Extract.only.tar.gz

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v176/Hydrus.Network.176.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v176.tar.gz

I had an excellent couple of weeks rewriting the gallery downloader page and the entire subscription system. I am glad I took the two weeks, as I only finished it yesterday afternoon. Your existing gallery pages and subs will be converted on update.

galleries

Gallery pages and subscriptions now use the new downloader engine. Gallery pages can be saved to gui sessions and should remember their file and tag options and also current search statuses, even down to exactly which gallery page they are currently on. Both also now have the new 'import status' button, so they recover much better from failed files and then let you retry them. In general, they load and save and work a bit faster. More information is displayed in a more efficient space.

Furthermore, gallery pages now support 'get tags even if file already in db' , and subscriptions are a bit easier to configure and have a new optional file limit on the number of files downloaded every time they run.

I am very pleased with the result here, but I have changed a lot of code. While I am fairly confident there is nothing very broken, there are likely a few bugs floating around. Please let me know if you notice anything not working how you expect!

In switching everything over, the Hentai Foundry filter options (where you can select the 'show yuri' filter and all that) proved a (continuing) pain in the neck to keep track of. I thought about writing a seperate class just for HF so I could remember this info and the respective tracking cookie, but I decided to keep the new code simple and non-hacky for now and made HF searches completely unfiltered and always sorted 'newest first', so they now always work like every other gallery hydrus supports. If people want content filtering, I plan to add tag-based filtering in future that should plug straight into the downloaders (so you can say "never download anything that has the tag 'blah'").

I may bring back hentai foundry 'popular/views first' sort for tag search if people want it, but each new sort will have to count as a separate gallery to select.

Also, I did not have time to convert 'default advanced tag options' over to the new system, so when you create a new gallery page or subscription, your import tag options will not be pre-checked.

full list

- the gallery downloader is updated to the new system

- the gallery downloader code is generally a bit better all around

- the gallery downloader will now remember its urls and file and tag options and paused status and so on throughout a reboot

- the gallery downloader supports 'get tags even if file already in db' option

- the gallery downloader management panel is updated for the new object. like the other panel updates, it is a little more minimalist

- the gallery downloader management panel now has the import status button so you can review your parsed urls and retry them, or copy errors, or whatever

- existing gallery downloader pages in gui sessions will semi-convert on the db update

- subscriptions are updated to the new system

- the subscription code is generally a bit better all around

- subscriptions will no longer dump out on a failed file!

- subscriptions can now recognise when files are uploaded during a page-walk (which shuffles everything up by one, of course), and recover without assuming it has synced

- subscriptions can now be set an initial file limit and a periodic file limit separately

- the manage subscriptions dialog is updated for the new object

- the manage subscriptions dialog now has the import status button so you can review your parsed urls and retry them, or copy errors, or whatever

- existing subscriptions will convert on the db update

- subscription popup now shows per-file download progress

- boorus that use number of thumbs to advance their gallery page indices can now resume

- the thread watcher now obeys 'exclude deleted' status more reliably

- the 'page of images' downloader now obeys 'exclude deleted' status more reliably

- subscriptions now attempt to recover their sync status when a new file is uploaded during sync

- fixed a bug where the thread watcher was not applying the filename namespace to new files, only redundant ones

- the old advanced_import_options dictionaries are all updated to the new import_file_options object

- improved subscription-dialog-was-opened-while-subs-were-running detection and reaction

- the way galleries are identified behind the scenes is now much simpler and more flexible

- for now, hentai foundry downloads and subs will no longer support the specific hf filter–they will be unfiltered and sorted newest first, so they work like all the other galleries. I may reintroduce 'sort by popularity' if there is demand for it

- a bit of importer status update refactoring and cleanup

- fixed some db vacuum maintenance timing that was interfering with gui reporting, so the 'currently vacuuming' popup never appeared until the vacuum was complete

- fixed a possible bug for import folders that have a local tag set

- splash screen is no longer stay_on_top–if there is demand, I will add an option

- fixed a bug in the page deleted detection code

- added page hidden detection code, replaced some ambiguous pause code with it

- improved pause/play code throughout the program

- corrected some bad x/y status texts

- for simplicity, the import_file db command now always returns ( status, hash )

- for simplicity, the import_file db command no longer takes service_keys_to_tags

- plenty of general code cleanup

next week

I will get 'default advanced/import tag options' working, and I already have plans to extend this new stuff to support a button for 'sync this sub on dialog ok', the ability to choose individual gallery streams (e.g. do you want just a Hentai Foundry artist's pictures or scraps, rather than both?), and adding a control for 'add tag x to service y for all files imported' to all instances of import tag options, which several people have asked for specifically for subscriptions.

Beyond that, I think all the import systems are converted to my new system, so that big job is completely done and I am free to think about a new big thing. First of all, I really want to rewrite the petition fetch/approve/deny system entirely, as I have let about 500 petitions pile up on my public tag repo because the current system has a terrible workflow. After that, I may look at adding audio support or moving towards mass 'looks very similar' dupe search, or adding text/html/shift jis file support, or just doing a whole load of small things. Let me know if you have something particular you would like!

I am behind on my messages and emails, so please bear with me for a couple of days while I catch up.

Anonymous 10/07/15 (Wed) 23:15:14 No.1191

just a heads up, default tag import options don't seem to work on the gallery downloader anymore.

So you have to manually check the boxes of the namespaces you want to import along with the files downloaded each time you open a gallery download tab.

Anonymous 10/08/15 (Thu) 01:33:41 No.1194

File: 1444268021114.png (37.21 KB, 885x383, 885:383, 15-10-08_12-30-40.png)

Guess this is what happens when you have about 15 pages open

repository synchronisation - _Dev - finished

0 updates downloaded, 3 updates processed, and 66,653 mappings added
exiting client
exiting db
hydrus client shut down at Thu Oct 08 12:04:23 2015
hydrus client started at Thu Oct 08 12:30:05 2015
booting client
booting db
updating db to v176
updating subscription makoto-mill
updating subscription tenryuu
saving updated subscriptions
updating gui sessions with gallery download pages
Traceback (most recent call last):
  File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.ClientController", line 864, in THREADBootEverything
  File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.ClientController", line 330, in InitModel
  File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.HydrusController", line 130, in InitModel
  File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.ClientController", line 47, in _InitDB
  File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.HydrusDB", line 69, in __init__
Exception: Updating the client db to version 176 caused this error:

Traceback (most recent call last):
  File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.HydrusDB", line 61, in __init__
  File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.ClientDB", line 6002, in _UpdateDB
  File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.ClientDB", line 5856, in ConvertSiteTypeQueryTypeToGalleryIdentifier
ValueError: too many values to unpack

A serious error occured while trying to start the program. Its traceback will be shown next. It should have also been written to client.log.
hydrus client shut down at Thu Oct 08 12:30:50 2015

Anonymous 10/08/15 (Thu) 02:48:25 No.1195

> subscriptions can now be set an initial file limit and a periodic file limit separately

So how does the system manage this exactly?

If it is set to 10 and there are 12 files in the initial sync, will it download 10 initially then next update get the remaining two? or just forget them?

As well will the periodic system ignore files when there are too many or is this just a way to limit downloaded images per time period, and download them later?

Anonymous 10/08/15 (Thu) 13:10:45 No.1196

>>1194

reverted, closed tabs and updated fine

Anonymous 10/08/15 (Thu) 13:26:16 No.1197

>>1196

Thanks for the info, had to do the same thing.

Anonymous 10/08/15 (Thu) 15:46:40 No.1198

>>1190

Thanks for the release!

>rewrite Petition fetch/approve/deny system

Yes please. Could you let anyone petition their own tags if done within a short time of upload? Helps with mistaken tags.

>mass 'looks very similar' dupe search

I've been waiting for this

Are there still plans to be able to import ratings (safe, questionable..) as tags to a rating namespace?

Anonymous 10/09/15 (Fri) 02:42:33 No.1199

Sometimes when I'm tagging files, I'll input a tag like "video games" but it appears in the tag list as "video g".

The first few times this happened, I thought it was just me hitting enter too soon, but I realized fairly quickly that Hydrus is clearly dropping characters off the end of what I enter.

Anonymous 10/09/15 (Fri) 06:31:26 No.1200

>>1199

I think you are pressing enter too quickly.

there's an autocomplete panel under what you type that takes about a second or two to catch up to what you've input in the text box.

When you press enter you don't confirm what you've typed, you select the autocomplete which is lagging behind you.

Anonymous 10/09/15 (Fri) 13:36:53 No.1202

>>1190

With the new update appears many errors. "Stop" button does not work, you need to close the tab. Import options are not set automatically, though they are in the settings tab "default tag import options". Well, can't upgrade if you have any open tab.

hydrus_dev ## Board Owner 10/10/15 (Sat) 18:50:30 No.1209

File: 1444503030685.jpg (1.13 MB, 1520x1000, 38:25, a692e3bfe2f6be20fc68d73cbb….jpg)

>>1191

Thank you. I am working on it this week.

>>1194

That's odd. It looks like you have a very old downloader page in one of your sessions, or otherwise a downloader page that has unusual site properties. I will revise my update code to delete pages it cannot parse. I am sorry for the inconvenience.

You can should be able to fix it now by booting in v175 and deleting the offending downloader pages in your gui sessions and then trying to update again.

Oh, I just read >>1196 and that's what you did. Great! Sorry again. As I was writing that update code I thought "Can this go wrong? Nah, it'll be fine." That's always the wrong thing to think.

>>1195

It is for the parsing step, where the sub discovers files. It will only add periodic_file_limit files per regular sync run. So, say you have a gallery search with three files:

XYZ

The sub will add up to initial_file_limit files from that search. Let's say the initial limit is 2, so it will add X and Y and then stop. (Otherwise, it will get everything).

Then it runs later. It recognises that it has run before, so it consults the periodic_file_limit. It gets a page of results:

TUV

If the periodic limit is 3, it will add T, U, and V and stop there. Otherwise, it will get the next page:

WXY

Where it will add W, and then see X and go "I saw that in a previous run. I must have caught up. I'll stop there."

I added periodic sync for people who want to get a sample of results from a big query, like "blue_eyes". To say, "Get ten files from that once a week, but no more."

I plan to add domain based bandwidth limits for the client in future, so you'll be able to say something like "Don't spend more than 100MB a week on my subs/this sub/this website."

>>1198

If someone petitions something they themselves added before the server creates the update file that would contain that content, it should be silently removed without the petiton being created. I am not sure if this actually works, so it is something I will be looking at in this rewrite.

Importing ratings is something I can add a lot easier now the downloader rewrite is done. It is in the to-do list.

>>1199

The exact workflow of the autocomplete dropdown can do with some work. When you hit enter, it usually actually selects whatever the listbox beneath the textbox thinks is selected, and during the short gui-freeze as db results are fetched or culled, this stuff can end up out of order. You might have typed what you wanted, but the other controls haven't caught up yet.

I plan to rewrite a/c results to stream in, which should improve the whole feel of this, but before then I will make sure to have another look at the order of the events here to stop your problem reoccuring. Thank you for reporting it.

>>1202

I hope to have default tag import options working for next week, and the upgrade problem should also be fixed by then. I will have a look at the stop button (I assume in the gallery downloader page), thank you for reporting the problem.

Anonymous 10/11/15 (Sun) 01:43:48 No.1212

>>1209

>The exact workflow of the autocomplete dropdown can do with some work. When you hit enter, it usually actually selects whatever the listbox beneath the textbox thinks is selected, and during the short gui-freeze as db results are fetched or culled, this stuff can end up out of order. You might have typed what you wanted, but the other controls haven't caught up yet. I plan to rewrite a/c results to stream in, which should improve the whole feel of this, but before then I will make sure to have another look at the order of the events here to stop your problem reoccuring. Thank you for reporting it.

It would be nice if the box could detect that what we've typed is a substring of a tag like it already does, but default to adding the existing tag when enter is pressed if there is only one tag that matches the substring.

If we want the substring as another different tag, put the substring in the box below the searched tag so we could double click that if we wanted, it just wouldn't be the default option.