win32
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v159/Hydrus.Network.159.-.Win32.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v159/Hydrus.Network.159.-.Win32.-.Installer.exe
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v159/Hydrus.Network.159.-.OS.X.-.App.dmg
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v159/Hydrus.Network.159.-.OS.X.-.Extract.only.tar.gz
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v159/Hydrus.Network.159.-.Linux.-.Executable.tar.gz
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v159.tar.gz
I had a very good couple of weeks. In case you missed it, a giant update was causing clients to crash from memory overload, so I shut down my server and put off last week's update to get a good fix done.
So, I have completely rewritten the update network protocol for hydrus, including how updates are generated serverside and stored to disk, to keep memory and cpu costs low. In my general testing, memory usage stays below 50MB (before, the problem 1.1 million tag update attempted to reserve many GB of memory and hence was crashing the client), and converting an update between a network string and a useful object is about two hundred times faster. Even better, I have saved about 30% on bandwidth.
Because of this important change, this release increments the hydrus network version, which means that v158 clients cannot talk to v159 servers and vice versa. I am turning my server back on today, and if you would like to keep syncing, you will have to update your client.
Furthermore, your client will have to redownload every update for every repository it syncs with. It will not have to reprocess those updates, only download them. For my public tag repo, it'll be about 220MB.
update rewrite
Essentially, I have done two things: I have changed my serialising library from YAML to JSON, and I have split large updates into smaller pieces.
YAML is a great protocol, and PyYAML is a great library that supports it, but it was not working for my purposes. It was really quite slow at doing what I wanted, and it did not allocate memory very efficiently as it parsed. I have scrapped it for the new JSON stuff I have been slowly moving to, which is reliably extremely fast and lean, and my new system allows for easier upgrading in future.
Very large updates are now split into blobs of about 100,000 'rows', where a row usually means something like a tag-file mapping. These sub-updates peak at anything from 1MB to 4MB, which is generally much easier and less laggy to deal with en masse than a giant 15MB+ chunk.
I am very glad I took the extra time to get this right. I would never have been able to get it out on Wednesday in any proper state.
I do not know how many people use hydrus, but I am obviously expecting a bit of a bandwidth rush with all the people who will be resyncing over the next week. If you encounter server errors, please pause your repo sync for a bit and try again later, and let me know what error text or delays or other experiences you have!
Furthermore, I have added a rows/s (i.e. rows per second) report to the normal update processing popup. I tend to get about 250 normally, and 1,000 or so on idle mode. I would be really interested in knowing what other values people get for this. I am also interested in knowing whether you are on a laptop or a pc, and whether it is a particularly powerful machine, whether you are on HDD or SDD, and whether the drive is encrypted or not.
tag export and import
I have ditched the old tag export and import YAML stuff for my new tag archives system.
If you want to export a bunch of tags to another client, just right click some thumbnails and select share->export->tags. This now creates a tag archive, which is just an efficient database file for storing tags.
You can simply import such a tag archive from the main gui's file menu.
4chan dumping disabled
I haven't touched my dumping system in a very long time. Some people told me it wasn't working, as 4chan's captcha has changed or something, and as I don't have time to fix it, I have disabled it. I would like to bring it back with better support for other imageboards.
idle mode
Usually after about twenty or thirty minutes of you not doing anything, the client turns on 'idle mode', which tells update processing to work in superspeed at the expense of making the gui laggy. You can now see when you are in idle mode from the status bar, and you can force it from the help->debug menu.
full list
- split previously monolithic repository updates into smaller pieces
- added service_update server calls
- extended content_update server calls to support sub-indices
- sped up some content update preprocessing code
- improved some content update preprocessing code
- radically reduced serverside memory usage while generating updates
- added iterator splitters to make sure any single update row cannot be too large
- thanks to iterator splitter, updates should process through the client more smoothly
- added timespan splitter to make sure any single server update query cannot be too large
- content updates are resumable if broken part way through downloading the list of them
- the update popup will state how fast it is currently working in rows per second committed
- cleanup of a lot of update related code
- more cleanup of update related code
- improved serialisable protocol so it'll work better over a network
- made serialisable protocol much simpler
- fixed numerical rating system predicate dialog slider range
- fixed numerical rating system predicate dialog OK for valued predicates
- fixed numerical rating custom filter action dialog
- improved some network yaml error handling
- replaced 'export tags' thumbnail menu entry with a tag archives system
- replaced 'import metadata' file menu entry with a tag archives system
- disabled thread dumper and manage imageboards and manage 4chan pass dialogs for now, because dumping code is out of date and completely screwed up
- client will now not start in idle mode
- help debug menu has new 'force idle mode' entry
- idle mode is displayed on status bar
- simplified client main gui status bar display and code
- the flash and flv files' embed button now has a little border to delineate it from the canvas background
- fixed some clientside bandwidth tracking code
- removed some old networking code
- made some custom objects draw themselves more efficiently
- for now, manage tag parents and siblings dialogs will not show deleted rows. I will eventually add a 'show deleted' checkbox, like the manage tags dialog has
- some static image rendering is slightly less laggy, particularly when browsing large images
- fixed initial height of manage tags dialog launched from preview window
- changed the 'search area' vs 'preview' sash gravity so the preview area won't expand or shrink on resize–see if you like it
- autocomplete dropdown should now hide itself when focus goes to other programs
- autocomplete dropdown should now hide itself when focus goes to other hydrus frames in os x
- fleshed out new URLCache object to handle better gallery download url management
- refactored some file status tracking stuff to a better system
- refactored gallery page fetching to a better system
- refactored gallery url handling to a better system
- some redundant import checking should be much faster
- improved error handling when booru image page parsing fails to find image url
- refactored how tags are fetched for DA, tumblr, and giphy as part of the above overhaul
- misc code cleanup
next week
I want to get back to adding Socks proxy support. I could have tried to do it yesterday, but I faced the same dilemma as last week, where I realised that the amount of work it will need means I should not rush it. So, I will do a generic network engine upgrade and then add proxy support, and then just work my list. I want to add those ratings colour options, improve the 8chan thread watcher for multiple files, fix some non-windows issues, continue a multi-month file downloading/importing rewrite, add a recycle bin service, and fifty other things.
With this semi-milestone release, I'd also like to generally say a renewed thank you to everyone for using my program. It is great hearing about the ways you use it and what you would like to see in future, and even though this recent problem took some time to fix, I really enjoyed the work.