Version 197

Name
Email
Subject
Comment *
File	Select/drop/paste files here
* = required field	[▶ Show post options & limits] Confused? See the FAQ.

Embed	(replaces files and can be used instead)
Options	Do not bump (you can also write sage in the email field) Spoiler images (this replaces the thumbnails of your images with question marks)
Password	(For file and post deletion.)
Allowed file types:jpg, jpeg, gif, png, webm, mp4, swf, pdf Max filesize is 8 MB. Max image dimensions are 10000 x 10000. You may upload 5 per post.

YouTube embed. Click thumbnail to play.

Version 197 hydrus_dev ## Board Owner 03/16/16 (Wed) 22:58:05 No.2274

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v197/Hydrus.Network.197.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v197/Hydrus.Network.197.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v197/Hydrus.Network.197.-.OS.X.-.App.dmg

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v197/Hydrus.Network.197.-.OS.X.-.Extract.only.tar.gz

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v197/Hydrus.Network.197.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v197.tar.gz

I had a great week. I fixed a bunch of bugs, and I managed to finish the first version of the first part of my new autocomplete cache layer. Depending on the size of your database and the speed of your computer, this week's update could take a little while, but once it is done, a lot of autocomplete results will appear very quickly, every time.

new cache layer

So, everything has generally gone to plan and even ahead of schedule, and I am overall really pleased. There's a new subfolder at install_dir/db/client_cache that will be populated by some new databases when you next boot the client. These databases cost a little extra CPU to maintain whenever new tags or files are added, but they can produce autocomplete results in a few milliseconds. Ultimately, the whole delay from sending the search string to the db to getting, sorting, filtering, and displaying the results typically maxes out at about 200ms. The part that was really lagging previously, where tags were being counted before being added to the old cache, which often took 9 seconds or so, now typically takes <5ms.

Ultimately, the biggest delay in autocomplete results for many people is now the artificial between-keypresses delay set in file->options->speed and memory, so make sure you check these options now and in the future as I roll out more of this new cache.

At the moment, this new cache supports autocomplete queries that have specific file services. Any autocomplete dropdown that has 'all known files' set (typically when you enter tags in the manage tags dialog) will use the old system, but anything else will search very quickly. Any search domains that include 'all known tags', are very fast but sometimes include counts that are too high.

The new cache databases are continually updated whenever you add or remove new files or tags. In my testing so far, I haven't noticed much slowdown in repository processing, but I would like to know if you notice your processing speed drops a great deal this week, or if perhaps importing some files or closing the manage tags dialog always produces a second's extra lag or similar.

The new caches have to sync up to the existing services, which basically means counting every single mapping by hand for each one. They try to do this efficiently by filtering only to pertinent file-mappings, but it can still take a while.

My fast i5 6400 dev computer with 6,000 files/20m mappings took about 2 minutes to update to v197, while my medium-fast i7 4700MQ IRL laptop with 300k files/50m mappings took 17 minutes. My 2.26GHz Core 2 Duo OS X testing laptop did a fairly more complicated db than my dev machine in 40 minutes, while my slow AMD E-450 Linux testing laptop is now on hour 2 as I write this. So: if you have a slow computer, prepare to wait a while for this week's update. Run a defrag beforehand if you know how, and make sure it isn't doing anything CPU or HDD heavy when you do update. Even if you have a fast machine, make sure you have a bit of padding in your schedule, just in case. Keep using v196 for a couple of days if you expect to have to shut your computer down or watch 1080p or play vidya or whatever.

If you don't sync with my PTR and don't otherwise have many millions of mappings, the update to v197 will probably take about five seconds.

Given the new cache's success, I will extend it to the 'all known files' service next, which shouldn't be too difficult, and then experiment with getting more accurate counts for 'all known tags', which may or may not prove worth it. There are also some database->maintenance->regen x ac_cache manual maintenance stuff I would like to add, in case anyone discovers a way these things can desync, and adding auto-vacuum and so on.

Caveat to all this: There is a lot of new code here, so let me know if you get wildly incorrect autocomplete results, or errors, or anything else you do not expect. I have seen some unreliable sibling behaviour for short exact-match sibling queries, so I will look into that, and I think the autocomplete result sorting is borked, although I can't remember if that is a new or old problem.

full list

- on client boot, autocomplete caches for specific file/tag service cross-references are now initialised and populated. progress is shown on the splash window

- on client boot, surplus autocomplete caches are deleted

- on service add, new autocomplete caches are created

- on file add/delete, autocomplete caches are updated

- on mappings pend/add/rescind pend/delete, autocomplete caches are updated

- the new autocomplete caches are consulted for all non-'all known tags' queries

- the old autocomplete cache no longer stores counts for specific file services, and the remaining associated maintenance calls are deleted

- databases now start their own mainloops

- databases now wait for their mainloops to finish prepping any large caches before they return to the controller

- the client database waits for autocomplete caches to finish before it finishes its own mainloop

- the padding around flash and the animation bar are included more accurately in some media zoom calculations, which should eliminate some general zoom jankiness and accidental 100% flash zoom coincidences that filled up the whole canvas

- fixed some clientside server boot error spam when local server or booru had no port set

- account refreshes that fail due to a network error will spam less to the log

- fixed .txt unicode tag parsing from import folders, which was not decoding at the correct step

- administrator immediate repository syncs now sync thumbnail downloads if needed

- service thumbnail sync will no longer superfluously check the presence of thumbnails whose files are local

- if a tag entered into the manage tags dialog has a sibling that already exists for all files, then a new 'ignore, as the sibling already exists' choice will appear

- fixed an overcounting bug in 'selection tags' when importing and adding tags at the same time

- fixed a typo in repository sync status text that was overcounting total number of updates by one

- fixed youtube downloader, which broke with the new library on my new dev machine

- the way that tags and predicates are filtered against a tag autocomplete text entry is now much faster

- bumped up the default content update chunk threshold from 100 rows to 5,000, which seems to be speeding up processing significantly, with a cost to recovery latency–see how it works for you

next week

Assuming this new cache doesn't explode for anyone, I'll go back to IPFS and try to get a directory parser/downloader working.

Anonymous 03/17/16 (Thu) 05:34:04 No.2275

It happened again:

2016/03/16 23:20:45: Trying to load tags from the .txt file "D:\toimport\boorutagparser-server-master\import_me\06b9c7ee66a9483e8a848a19f0335545.png.txt" in the import folder "jetboom" threw an error!

2016/03/16 23:20:45: Exception:

2016/03/16 23:20:45: NameError

global name 'tags' is not defined

Traceback (most recent call last):

File "include\ClientImporting.py", line 1119, in DoWork

service_keys_to_tags = { service_key : tags for service_key in self._txt_parse_tag_service_keys }

File "include\ClientImporting.py", line 1119, in <dictcomp>

service_keys_to_tags = { service_key : tags for service_key in self._txt_parse_tag_service_keys }

NameError: global name 'tags' is not defined

While trying to import a folder full of .txt files. I've looked through some of the txt files the logs mention, but I'm not seeing any strange characters that might be causing this.

The images themselves imported fine, but around half of them didn't import their tags.

I redid the import manually, and it worked fine. I'm not entirely sure how to recreate this bug

Anonymous 03/19/16 (Sat) 10:41:34 No.2284

>>2275

I'm getting this error too, when hydro is importing from my automatic import folder.

No errors if I do it manually through file->import files

hydrus_dev ## Board Owner 03/19/16 (Sat) 19:12:42 No.2285

File: 1458414762701.jpg (406.81 KB, 1280x996, 320:249, d6a97887d284ac84786c27f743….jpg)

>>2275

>>2284

I apologise–I did test this, but I went back later to clean up the code a little and made a typo that somehow slipped through my normal final checking routine. I have fixed it for next week.

Anonymous 03/20/16 (Sun) 05:20:18 No.2289

Is there a way to control the order tags are listed in the autocomplete section of the 'manage tags' dialogue?

It was either because of the latest update or some setting I unwittingly changed, but there doesn't seem to be a rhyme or reason to the ordering anymore, which makes the autocomplete fairly useless at the moment

hydrus_dev ## Board Owner 03/20/16 (Sun) 17:30:02 No.2292

File: 1458495003102.jpg (1.61 MB, 1800x1362, 300:227, 84cab086dc51acbb56ffc07eee….jpg)

>>2289

I accidentally broke it in this release–it is fixed for next week! Some sibling replacement stuff should work better as well!

Anonymous 03/21/16 (Mon) 19:03:55 No.2298

Is it just me or is the upnp broken?

hydrus_dev ## Board Owner 03/22/16 (Tue) 16:47:25 No.2305

File: 1458665245458.jpg (314.79 KB, 1837x1139, 1837:1139, 500fc0e0bc2e06a54ad738558c….jpg)

>>2298

It seems to be generally working for me. Which part is giving you problems? Do you have an error message?

Anonymous 03/22/16 (Tue) 19:44:30 No.2308

>>2305


2016/03/21 20:27:09: Daemon UPnP encountered an exception:
2016/03/21 20:27:09: Exception:
2016/03/21 20:27:09: Exception

Problem while trying to add UPnP mapping:



upnpc : miniupnpc library test client. (c) 2005-2013 Thomas Bernard

Go to http://miniupnp.free.fr/ or http://miniupnp.tuxfamily.org/

for more information.

List of UPNP devices found on the network :

 desc: http://192.168.1.1:5000/rootDesc.xml

 st: urn:schemas-upnp-org:device:InternetGatewayDevice:1



Found valid IGD : http://192.168.1.1:5000/ctl/IPConn

Local LAN ip address : 192.168.1.247

ExternalIPAddress = 10.5.131.97

AddPortMapping(25510, 25510, 192.168.56.1) failed with code 718 (ConflictInMappingEntry)

GetSpecificPortMappingEntry() failed with code 714 (NoSuchEntryInArray)



Traceback (most recent call last):
  File "include\HydrusThreading.py", line 218, in run
    self._callable( self._controller )
  File "include\ClientDaemons.py", line 399, in DAEMONUPnP
    HydrusNATPunch.AddUPnPMapping( local_ip, internal_port, external_port, protocol, description, duration = duration )
  File "include\HydrusNATPunch.py", line 84, in AddUPnPMapping
    raise Exception( 'Problem while trying to add UPnP mapping:' + os.linesep * 2 + HydrusData.ToUnicode( output ) )