Version 198
windows
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v198/Hydrus.Network.198.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v198.tar.gz
I had a good week. I added IPFS directory downloads and sped up file search result generation.
IPFS directories
If the IPFS downloader receives a directory multihash, it now asks IPFS for the full nested directory listing and throws that up on a new checkbox tree dialog. You select the files and folders you want to download, hit ok, and the IPFS downloader should attempt to get them all.
The new tree dialog is something new, and I do not know if it renders well on all platforms, so let me know if you have any problems. The overall workflow of this is also fairly rough–I expect I might eventually move this to a traditional queued download page, to support resumes, retries and better error reporting and so on.
The last thing I would like to do to round off 'IPFS plugin v1.0' is adding directory upload. Assuming I can figure out a nice way to do it, preferably with the API at port 5001, then I assume that will be done in the next few weeks.
faster file search results
One particular query inside file search result generation (where it says '256 of 2,000 files…' in the status bar) had become incredibly slow. I am not sure when it got so bad, nor exactly what the problem was (the sqlite query optimiser reported no problems, as far as I understand it), but I have slightly reworded it and the lag has disappeared. On my dev machine, the query went from taking 2.5 seconds per 256 files to 9 milliseconds! Anyone with a lot of tags will see a similar boost in speed when loading files or opening the manage tags dialog, which happens to use the same routine.
full list
- ipfs download now parses directory multihashes
- ipfs directory trees are thrown up on a new checkbox tree dialog for selection/filtering
- created a new download_urls (i.e. multiple urls in one popup) popup call, which the new ipfs directory downloader uses
- improved a very inefficient line of sql in the mappings fetch stage of query result building that was also used in the manage tags dialog. on my dev machine, fetching 256 files' mappings dropped from 2.5s to 9ms!
- regular gallery pages and 'page of images' pages now have a 'paste input' button to mass-add newline-separated queries from the clipboard
- fixed a typo in new txt_tags import folder code
- fixed bad autocomplete predicate sorting
- fixed some sibling autocomplete search_text matching logic
- some byte-based value/range presentation will now correctly convert to bytes, rather than regular decimal numbers
- improved linesep splitting code across the program
- fixed get/setlabel calls across the program to get/setlabeltext, so ampersand characters are handled properly
- wrote an 'all known files' autocomplete cache db and prepped a bunch of code for it–it should be easy to finish next week
- added a new regex practise link to the regex dialogs
- os x app release should display better on retina displays
- misc cleanup
next week
I wrote most of the 'all known files' autocomplete cache this week, so I will polish that off and then focus on some smaller stuff, which has been piling up.
Q&A thread: For simple questions that don't need their own thread
I feel like we need a single thread to contain all the single-line answer questions that are asked around here. Would provide a nice archive of "noob questions", and would prevent the board from being clogged up by simple, single-purpose Q&A threads.
Suggestions
Drag and drop windows with tag rules. Show two windows side by side and one window can be programmed with the rule "ADD tag foo" and the other one has the rule "REMOVE tag foo, ADD tag bar" and you can drag and drop files to them.
Deriving tags from regex of other tags/namespace tags. A file has the tag "filename:big_ugly_name" and we could regex that namespace for another tag.
Tag sets with hotkeys: save a set of tags under a hotkey so it's quick to add them to a file while filtering
Opaque window behind tag list in the corner so it doesn't get hidden by picture background
Option to default certain mime types to be excluded from slideshow and only open externally, will help with videos with odd codecs that don't preview in the slideshow correctly
Option to specify hamming distance in "find similar images", you can't change the option once it's in the filter window and you have to enter the hash manually in the "system:similar to" option
Bugs
I suppose I'll start a thread for bug reports?Description
Hydrus fails to upload the petition for files if files were queued only for deletion on the server. This does not happen if files are both queued for uploading and some are queued for deletion.
Basically boils down to files pending count being anything like (0/x) failing and working if (>0/x).
Pic related.
Traceback
UnboundLocalError
local variable 'i' referenced before assignment
Traceback (most recent call last):
File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.HydrusThreading", line 163, in run
File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.ClientGUI", line 235, in _THREADUploadPending
UnboundLocalError: local variable 'i' referenced before assignment
Tag parser script
Here's something I did to make copying individual booru tags easier. hydrus is good with booru browser but its page downloader doesn't read tags. And sometimes you just wanna quickly add tags to an individual image.
So this tamper/greasemonkey script will add a button and a key bind to booru posts and the Illustration2Vec demo site to copy all tags on the page including their schemas and rating. Then you can just right click your image in hydrus > manage file tags > paste tags > apply. The whole process takes about 5 seconds.
github.com/JetBoom/boorutagparser
Release Tomorrow!
I had a good week. I fixed some bugs, added IPFS directory parsing and download, and massively sped up file query results generation.
The release should be as normal tomorrow.
Loading speed & mac os x question
Hello,
first of all: neat program! :) Really like the features. I've been looking for a suitable application for collecting stuff such as memes and artwork forever. I'll probably also use it for my No Man's Sky screenshots when it's finally out. I'm expecting to take a shit ton and to tag them with all the relevant data like planet name, available resources, creatures etc. so I can look it up later if needed or just for the memories - since we don't really know how much the ingame encyclopedia will track.
Anyway, I've connected the hydrus network client to the remote tag repository and updated all the way. I've then imported just a single folder of 40 quite popular images (ranging from around 500kb to 5mb) to see if the auto-tagging is working. Works like a charm. All the picturs are sitting at around 50 tags each now.
One question though: is it normal that loading the images via system:everything takes around 20 seconds (used my phone timer to measure)? Remember, I do only have 40 pictures in the database (and all the tag mappings from the repo, but coming from a dev background myself I'd think think they'd be ignored in the query when using system:everything). Using the title:xyz tag to limit the result set to a single image also takes around 17 seconds. Normal?
I'm sitting behind a fairly powerful gaming pc that's also kept very clean and has no performance issues in any other application. Relevant specs: i5 4690k, 32 gb ddr3 2400mhz, 1tb samsung 850 pro series ssd (yes, my only drive, I only care for the performance and don't need that much space anyway :P). I guess my gtx 980 ti won't matter much in this case. :D
I've already tried vacuuming the db. Any suggestions or feedback?
—
Second question: I'm also using a macbook pro with a high-dpi display I'd also like to run the client on (probably syncing the db between the two devices). It seems the client doesn't support the doubled resolution of its so-called retina display (it has a 2880x1800 resolution but is basically rendering it as it would a 1440x900.
using the quadrupled pixels for extra sharpness), so everything looks quite pixelated. Doesn't matter in the application but this unfortunately also applies to images. I guess there's no easy way to fix that (maybe a limitation of the python gui renderer used)?
Version 197
windows
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v197/Hydrus.Network.197.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v197.tar.gz
I had a great week. I fixed a bunch of bugs, and I managed to finish the first version of the first part of my new autocomplete cache layer. Depending on the size of your database and the speed of your computer, this week's update could take a little while, but once it is done, a lot of autocomplete results will appear very quickly, every time.
new cache layer
So, everything has generally gone to plan and even ahead of schedule, and I am overall really pleased. There's a new subfolder at install_dir/db/client_cache that will be populated by some new databases when you next boot the client. These databases cost a little extra CPU to maintain whenever new tags or files are added, but they can produce autocomplete results in a few milliseconds. Ultimately, the whole delay from sending the search string to the db to getting, sorting, filtering, and displaying the results typically maxes out at about 200ms. The part that was really lagging previously, where tags were being counted before being added to the old cache, which often took 9 seconds or so, now typically takes <5ms.
Ultimately, the biggest delay in autocomplete results for many people is now the artificial between-keypresses delay set in file->options->speed and memory, so make sure you check these options now and in the future as I roll out more of this new cache.
At the moment, this new cache supports autocomplete queries that have specific file services. Any autocomplete dropdown that has 'all known files' set (typically when you enter tags in the manage tags dialog) will use the old system, but anything else will search very quickly. Any search domains that include 'all known tags', are very fast but sometimes include counts that are too high.
The new cache databases are continually updated whenever you add or remove new files or tags. In my testing so far, I haven't noticed much slowdown in repository processing, but I would like to know if you notice your processing speed drops a great deal this week, or if perhaps importing some files or closing the manage tags dialog always produces a second's extra lag or similar.
The new caches have to sync up to the existing services, which basically means counting every single mapping by hand for each one. They try to do this efficiently by filtering only to pertinent file-mappings, but it can still take a while.
My fast i5 6400 dev computer with 6,000 files/20m mappings took about 2 minutes to update to v197, while my medium-fast i7 4700MQ IRL laptop with 300k files/50m mappings took 17 minutes. My 2.26GHz Core 2 Duo OS X testing laptop did a fairly more complicated db than my dev machine in 40 minutes, while my slow AMD E-450 Linux testing laptop is now on hour 2 as I write this. So: if you have a slow computer, prepare to wait a while for this week's update. Run a defrag beforehand if you know how, and make sure it isn't doing anything CPU or HDD heavy when you do update. Even if you have a fast machine, make sure you have a bit of padding in your schedule, just in case. Keep using v196 for a couple of days if you expect to have to shut your computer down or watch 1080p or play vidya or whatever.
If you don't sync with my PTR and don't otherwise have many millions of mappings, the update to v197 will probably take about five seconds.
Given the new cache's success, I will extend it to the 'all known files' service next, which shouldn't be too difficult, and then experiment with getting more accurate counts for 'all known tags', which may or may not prove worth it. There are also some database->maintenance->regen x ac_cache manual maintenance stuff I would like to add, in case anyone discovers a way these things can desync, and adding auto-vacuum and so on.
Caveat to all this: There is a lot of new code here, so let me know if you get wildly incorrect autocomplete results, or errors, or anything else you do not expect. I have seen some unreliable sibling behaviour for short exact-match sibling queries, so I will look into that, and I think the autocomplete result sorting is borked, although I can't remember if that is a new or old problem.
full list
- on client boot, autocomplete caches for specific file/tag service cross-references are now initialised and populated. progress is shown on the splash window
- on client boot, surplus autocomplete caches are deleted
- on service add, new autocomplete caches are created
- on file add/delete, autocomplete caches are updated
- on mappings pend/add/rescind pend/delete, autocomplete caches are updated
- the new autocomplete caches are consulted for all non-'all known tags' queries
- the old autocomplete cache no longer stores counts for specific file services, and the remaining associated maintenance calls are deleted
- databases now start their own mainloops
- databases now wait for their mainloops to finish prepping any large caches before they return to the controller
- the client database waits for autocomplete caches to finish before it finishes its own mainloop
- the padding around flash and the animation bar are included more accurately in some media zoom calculations, which should eliminate some general zoom jankiness and accidental 100% flash zoom coincidences that filled up the whole canvas
- fixed some clientside server boot error spam when local server or booru had no port set
- account refreshes that fail due to a network error will spam less to the log
- fixed .txt unicode tag parsing from import folders, which was not decoding at the correct step
- administrator immediate repository syncs now sync thumbnail downloads if needed
- service thumbnail sync will no longer superfluously check the presence of thumbnails whose files are local
- if a tag entered into the manage tags dialog has a sibling that already exists for all files, then a new 'ignore, as the sibling already exists' choice will appear
- fixed an overcounting bug in 'selection tags' when importing and adding tags at the same time
- fixed a typo in repository sync status text that was overcounting total number of updates by one
- fixed youtube downloader, which broke with the new library on my new dev machine
- the way that tags and predicates are filtered against a tag autocomplete text entry is now much faster
- bumped up the default content update chunk threshold from 100 rows to 5,000, which seems to be speeding up processing significantly, with a cost to recovery latency–see how it works for you
next week
Assuming this new cache doesn't explode for anyone, I'll go back to IPFS and try to get a directory parser/downloader working.
Bare Client Database with PTR Sync Initialised
Here is a 7zip of the client database, version 147, freshly initialised and synced up to my public tag repository as of today.http://www.mediafire.com/download/0nh9z1994iao4jl/Hydrus_Network_Client_bare_database_with_PTR_up_to_update_1012.7z
If you want to start a new client that connects to my public tag repository, you can swap in this database right after you install, and you won't have to spend twenty hours sitting around waiting for 7.8 million mappings to process.
If you have no idea what this is, I suggest you ignore it and install the client normally, learning about how hydrus works using my help files first.
Release Tomorrow!
I had a great week. I fixed a lot of bugs, and I managed to finish the first part of my new autocomplete tags cache. I still need to test it on a real-world database, and the cache does not yet cover all search domains, but in general it seems to be producing autocomplete results for a 6,000 file/20m mappings db in about 40-100ms. I am obviously extremely pleased. The cache takes a bit of CPU to initialise and sync with an existing service, so I expect the update to take a few minutes, possibly longer.
Absent any problems in my extended testing, the release should be as normal tomorrow.
A local booru program is exactly what I've been looking for! Thank you Hydev.
One q: Is it possible to take your local images, compare them to a booru's images, and apply all tags that the booru has for the images to your local ones?
Even if it's time-consuming, it would make my life a ton easier.
Timestamp namespace
I think it would be worth it to have a proper timestamp namespace with some usability features:
Options to pull stamp from the filename (most image boards use the Unix timestamp for the filename) otherwise optionally use the file modified time from the filesystem
Autocompletion for missing time fields, e.g. if only a year and month are entered (publication date) then the rest of the timestamp is set to midnight on the first of the month or other option
Version 196
windows
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v196/Hydrus.Network.196.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v196.tar.gz
I had a good week. I fixed and improved some things, and I made some important changes to the autocomplete tag code.
The past couple of weeks have brought a lot of new mappings to my public tag repository, and I think we yesterday hit 50 million! It represents about 700k different tags applied to 3 million files! I'm really pleased, and I appreciate the contributions everyone has made. While this expansion is revealing some terrible lag in places, I still think it neat that we have collectively gathered so much stuff.
autocomplete rewrite
I made good progress this week on the new autocomplete cache layer I have been planning. I have a good plan of where I am headed, a semi-fleshed skeleton of the cache database written, and a fair amount of optimism about bringing it all together. I am generally confident the new cache will deliver sub-second autocomplete results for pretty much all queries when it is done. I still need to think about some things, and there is a lot of work still to do, but I've made some decisions on what I can and cannot easily attempt to calculate in quick time. Disabling 'all known files'/'all known tags' last week was part of that.
I also gave the existing code a polish and converted it to this new calculation philosophy. Some sibling stuff is fixed, and some bits of inefficient code are cleaned up, but the important thing is that 'all known tags' autocomplete results (e.g. the sort you get when you open a new local search page and start typing) are calculated in a faster but sometimes less accurate way. If you only have 'local tags' and my PTR, then you will likely see no change, but if you have multiple services that contain the same tags for the same files, you will see counts that are the sum of those services' tag counts, rather than their union (they will be a bit higher than is accurate, sometimes).
I am not sure if I will be able to bring back 100% accurate counts for absolutely all cases, but I will revisit it after the first version of the cache layer is done. I might end up including it in a second layer that collapses sibling counts and applies local censorship rules as well, but we will see. For now, I plan to add '<=' signs to uncertain counts and see how much of a big deal it turns out to be.
Anyway, the upshot of changing this calculation is that all autocomplete results seem to be returning a bit quicker than usual already. That still often means ten seconds, so the new cache is still needed, but at least we are headed in the right direction. Regular tag processing, as well, which interacts with the existing cache, seems to be running up to 40% faster, and some other big jobs like resetting services now no longer have to do a very laggy autocomplete purge. Let me know how you get on!
A lot of orphaned cache entries need to be deleted, so the update may take a few minutes.
other stuff
The 8chan thread watcher should now work for all boards. Let me know if it doesn't work for something unusual.
I also fixed the tumblr downloader, which recently broke due to a subtle API change.
The IPFS downloader now detects directory multihashes and gives a more pleasant error. I will next add directory downloading and then directory pinning.
The recent 'load tags from neighbouring .txt files' import option is now available for import folders. If you are interesting in this, check it out at the edit import folders dialog and let me know if the new option makes sense. The .txt files should be deleted or moved alongside the media files they refer to depending on how you have those actions set up.
full list
- fixed the 8chan thread watcher for boards that host content on media.8ch.net
- improved the thread watcher url check logic so it won't lag with the new fix
- cleaned up the ac generation code a little
- 'all known tags' ac counts are now summed from all the known tag services rather than calculated directly (a <= indicator for when these cases overlap will be forthcoming). this speeds up file add/delete, service reset, a/c fetch time, and general tag processing, and reduces the size of the db
- ac generation code now deals with 'is the entry text an exact match or not?' better
- ac generation code will now no longer produce non-exact-match siblings on an exact match search
- ac generation code will no longer save half complete search text into the db as new tags
- on update, the a/c cache and its helper table 'existing tags' will be cleaned of a lot of orphans, which may take a few minutes
- fixed some bad unicode path parsing when importing files in some OSes, I think!
- fixed some bad read autocomplete sibling substitution
- fixed a bug where autocomplete predicate lists would not update if the new list was merely a reorder (which can happen in some unusual sibling cases)
- fixed the tumblr parser for the subtly new API
- import folders now support loading tags from neighbouring .txt files–check the dialog to set up which tag services you would like to import to
- the ipfs file downloader now queries DAG object links, determines if the given multihash is a directory or other complicated object, and if so politely dumps out (handling of directory downloads is forthcoming)
- some db code is cleaned up
- prepared db code for some future subclasses
- wrote most of the new ac cache db
- misc cleanup
- added some browser addon links to the ipfs help
next week
I will continue this cache layer stuff, and I'll also see about IPFS directory parsing and a bit of gui to pick the files you want to download.
My overall near-future 'big stuff' plan is to go something like IPFS directories->cache layer->extract service data from client.db->suggested tags control->faster dupe searching->make a new poll on what to work on next.
Tag hierarchy
I've been trying to use tags with hierarchy which lets subtags describe the tags above them, using periods as delimiters.
hair
hair.black
hair.long
hair.long.knee length
hair.style.braids
hair.style.ribbon
This is nice for keeping relevant tags close and finding characteristics of tags that I want to describe in more detail.
The problem is searching. If I try to search "braid" it doesn't hit on "hair.style.braids" and if I search *braid it takes a very long time, along with the danger of the autocomplete taking off with searching for the asterisk and locking up the client for several minutes.
I've been able to work around this a little bit with siblings and searching for "braided hair" -> "hair.style.braids".
Can you think of a way to start searches inside tags following some kind of delimiter like a period?
Release Tomorrow!
I had a good week. I fixed a lot of stuff, including the 8chan thread watcher and the tumblr downloader, added the new .txt tag parsing to import folders, and significantly improved parts of the tag autocomplete code in preparation for the new a/c cache layer.
The release should be as normal tomorrow.
Version 194
windows
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v194/Hydrus.Network.194.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v194.tar.gz
I had a good week. I mostly worked on IPFS, and successfully got pin and unpin working, and otherwise did a bit of cleaning up and added a new way to add tags on import.
ipfs pin/unpin
Now that I know this all works, I will write up a help page for IPFS, including what it even is and where to get it, for next week, so if you don't understand any of this, please hang on a little bit.
If you have an IPFS service added, right-clicking on thumbnails will give you the option to remote services->pin to ipfs. The workflow works just like hydrus repositories, so your 'to be pinned' files will gain a little icon and the pending menu will count them. Committing from the pending menu will upload them to your IPFS daemon when you are ready. Files that are pinned to IPFS will retain a small correct-colour IPFS icon. Unpinning works similarly.
Furthermore, right-clicking on pinned files will let you copy their IPFS multihash, so you can share them wherever you like. Here's some I did this week:
QmP6BNvWfkNf74bY3q1ohtDZ9gAmss4LAjuFhqpDPQNm1S
QmXH3NYyqyAnNgf9gWppufJp3kCHwJcJeqk2w3d9zQQF9u
QmaQNZLHQg3tXT2R3gCSKPDJ4Bs9FjmWnHbmNQbVS157ge
You can try to download and import those through the client, or you can check through the browser-compatible gateway the IPFS people run at:
https://ipfs.io/ipfs/[multihash]
I haven't set my dev machine's IPFS daemon to run all the time, but I presume the file will live on despite that, somewhat, precisely because of how IPFS works. I have read all three files through that gateway, so I think they get temporarily mirrored or something. Let me know if you can get them, and let me know how your own IPFS-hydrus stuff goes!
parse tags from neighbouring .txt files
For a while now, when you export files from the thumbnail right-click menu via share->export->files, you have been able to export the files' tags into neighbouring [exported_filename].txt files, as newline-separated lists. I have now added a checkbox to the import files 'path tagging' dialog (the one with all the regex stuff), that will try to do the reverse, loading any existing neighbouring .txt files and parsing them for newline-separated tags.
If you have been looking for a way to automate tag import from an unusual source with a non-python script, have a play with this!
full list
- ipfs pins and unpins can now be queued up like file repository pending and petitioned, through the regular thumbnail right-click menu, which also reports some/all ipfs pinned selection status
- this ipfs action queue is similarly summarised and commited at the normal service 'pending' menu
- ipfs's 'pinned', 'to pin', and 'to unpin' statuses are displayed on thumbnails with ipfs-specific icons
- you can copy the focussed file's ipfs multihash or all the selected files' ipfs multihashes from the thumbnail menu's share->copy->ipfs multihash
- added a .txt tag parser to the 'path tagging' import dialog–it will parse the same sort of txt files the export dialog produces
- the client's new 'requests' network code is harmonised, generally improved, and now produces hydrus-compatible exceptions
- updated help re the local server and boorus now defaulting to off
- db can now remember service-specific filenames (e.g. ipfs multihashes)
- cleaned up some overly complicated and confused thumbnail menu code
- the pending menu now specifies what it is about to do more plainly
next week
I want to support IPFS directory sharing, so you'll be able to post 'Here's my 400-file blah collection: [single_multihash]' to a thread, and also be able to efficiently download such links using hydrus. It needs a bit more research on my end and thinking about what workflows would be best and then experimenting with the IPFS API to figure out what is actually doable. That won't happen in one week, because as well as the help, I also want to add 'system:pinned to ipfs' search support and a better review services panel for IPFS.
I might also write some class outlines for the big client.db extraction.
Importing danbooru/gelbooru db
Can a kind sould point me out how to import the danbooru/gelbooru tags? When I go to file/import metadata all I get is this error:'utf8' codec can't decode byte #xea: invalid continuation byte
in "C:\Hydrus Network\db\danbooru.db", position 31
Version 195
windows
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v195/Hydrus.Network.195.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v195.tar.gz
I had a good week. I worked on several different things and did a bit more IPFS stuff. An important bug fix means all clients will have to do a bit of reprocessing.
ipfs stuff
I've written up some basic IPFS help here:
https://hydrusnetwork.github.io/hydrus/help/getting_started_ipfs.html
So if you have been holding back, check that out and have a play with it.
I've also added IPFS services to 'system:file service', so you can now find all your pinned files again, and I've added an option to the manage services dialog to make copying multihashes more flexible.
repository processing bug
I discovered an important bug in repository processing that meant some content was not being processed in certain situations. I have fixed the bug, and I've retroactively semi-reset repository processing. Your repositories will go over all their updates again and reprocess, picking up anything that was missed.
Even though the clients are going back to the start, it won't take too long to catch back, as reprocessing content that is already in the db works about 10-20 times faster than regular processing (my laptop is working at 8,000 rows/s, my dev machine at 20,000 rows/s). Just let your clients keep syncing like normal, and you should be caught up again within the week.
8chan thread watcher
I fixed a bug that was sometimes missing OP images in the 8chan thread watcher, but I've since discovered a new problem–lots of boards have started to host their images and thumbnails off media.8ch.net rather than 8ch.net (including now /hydrus/). I presume this is related to the various dev work and migrations and so on of the recent months being sorted out, but unfortunately the JSON API doesn't specify which domain I should be using, or at least I can't find where it so says.
I don't follow 8chan news very closely, so have I missed some tweet about a migration to media.8ch.net for content hosting? Is there a master board list somewhere of which boards are using which domain that I can just hardcode for now? Is there a 'board API' that'll give a bit of JSON describing the board? Something like this:
https://8ch.net/settings.php?board=k
If there is an easy answer to this that I have missed, let me know, otherwise I'll go officially ask someone. I guess /operate/ would be the right place?
If there isn't an easy answer, then I'll probably write an extra step into the watcher that fetches the html version of the page and inspects the links manually to figure out the domain, and cache that info for the rest of the session.
Anyway, for now I am keeping the thread watcher pointed at 8ch.net. /pol/ and /v/ and a handful of others work, but /s/ and and /k/ don't, for instance.
all known files and tags
In preparation for the new cache layer, I have disabled searches over 'all known files' cross-referenced with 'all known tags'. This search domain is so massive, computationally heavy, and so very rarely used, that it is ultimately best to just not support it going forward. If you want to search absolutely everything your client knows about, open up several 'all known files'/'specific tag service' searches in parallel.
full list
- wrote up a v1.0 ipfs help page
- added ipfs services to 'system:file service'
- added a 'multihash prefix' option to ipfs services in 'services->manage services'–it will prefix thu given text to multihashes copied to the clipboard (e.g. you could use 'http://127.0.0.1:8080/ipfs/')
- fixed a bad repo sync processing bug that was incorrectly assuming packages were completely processed when they often were not
- all repos will have their processing timestamps reset on update, forcing a (fast) reprocess of everything they have to cover the previously missed content
- fixed 8chan OP image parsing, which was sometimes failing due to absent md5s in the json
- widened the splash screen a little more so repo sync text can fit a bit better
- a couple of splash status texts are shuffled around
- updated to sqlite 3.11.0 for windows
- if sqlite3 reports at least version 3.11.0, the db will stick to WAL for large transactions (lke vacuum and content processing), since these are fast now
- 'namespace:*anything*' will now only appear if the a/c input has no 'subtag' component., e.g. 'character:'
- import and export folders now print simple summaries to the log if they do some work
- tidied up the 'pages' menu
- the various hardcoded 'delete' key events are now also triggered by a backspace on os x
- added a rough 'copy known urls' to the regular thumbnail menu. this is prototype, let me know how you would like this information managed and displayed in future
- in prep for the new cache layer, the autocomplete dropdowns (and hence search domains of all contexts) no longer support 'all known files' cross referenced with 'all known tags'
- misc cleanup
next week
I'll move both IPFS directory sharing and my new cache layer forward.
Release Tomorrow!
I had a good week. I improved several miscellaneous things, fixed some bugs, including a bad flaw in repository processing, and added a couple of IPFS bells and whistles, including a new help page.
The release should be as normal tomorrow.
Version 193
windows
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v193/Hydrus.Network.193.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v193.tar.gz
I had a good week. I am feeling much better and am generally back to schedule. I fixed a lot of bugs and improved several different things. Unfortunately, my plans for IPFS integration encountered a small snag.
Incidentally, I played through Human Resource Machine this week and really enjoyed it. If you are a software engineer, or are looking to become one, check it out.
the client's servers are off by default now
You can now switch off the client's local server under file->options->local server. You may not have heard of this server before–it is a mostly experimental http server that was useful for exporting files in earlier versions. I will do more with it in future, but regular users don't need it now we have drag-and-drop export.
The same is true for the local booru, under services->manage services->local booru.
For a new client, these servers now start off. I never liked the 'welcome to hydrus, now your firewall will ping' message I started with, especially when most people didn't even need those servers, so that's also now gone as a result.
ipfs
I wanted to add simple hydrus->ipfs file upload this week, but in working on that I discovered that I had misunderstood what an ipfs multihash (the Qm… address) represented. I thought it was the sha256 of the file contents, but it is actually the sha256 of an internal ipfs-specific data node that leads to and eventually becomes the whole file. I am not 100% familiar with IPFS's inner workings, but I think I appreciate why they do it this way.
Unfortunately, I had hoped and assumed I could nicely overlap hydrus's cache of sha256 hashes with their file list and hence easily sync and push and interpolate and query that metadata over the IPFS API. But it seems that figuring out what an IPFS multihash for a particular file isn't trivial. This simply adds a layer of complexity to anything I want to do, as I'll have to cache sha256->ipfs_multihash pairs and won't be able to generally query the IPFS network for arbitrary files.
So, I will keep working on IPFS, with the aim of getting file upload and virtual 'directory' upload working (so you can IM your friend the IPFS multihash, or post it on a board somewhere, and hence do ad-hoc p2p), and pause there. This was generally my plan anyway, but some of the fun speculative things we were previously discussing just might not be technically possible. Distributing repository update files and doing other client-to-client supplemented network comms is still doable, and something I will return to as it becomes necessary.
changes in how jobs are scheduled
I am generally dissatisfied with how slow some of the client's operations have become. Some big jobs trigger when I don't want them to, or don't when I do, and some simple tag autocomplete queries take just too long. A lot of this code is messy.
So, I have spent a bit of time improving how these things trigger in real world scenarios. Let me know if anything bad or good happens. Particularly, big jobs should cancel more quickly (especially on slower computers), and subscriptions should trigger much more snappily now, including just a couple of seconds after client boot. If that turns out to be annoying, let me know.
I want to do more here, in particular serialising big jobs to a single daemon thread and giving the user more power to say 'sync this repo now'-type stuff. I also have an idea to break the main client.db into smaller files and 'service' folders that will be quicker to maintain. The service dbs will handle their own sync, not locking up everything else while they do so, and you'll be able to delete them and export and import them to other clients without having to do a ton of reprocessing. (Hence I'll be able to put up a synced ptr .7z that anyone can import.)
thoughts on speeding up tag autocomplete results
The current tag autocomplete system worked great when there were only 500,000 tags, but it really groans now. I have optimised and optimised, but some queries still just take too long.
I know what the problem is, and I know the current system won't ever be able to produce accurate results quickly, so I want to take a little time in the next few weeks–probably as part of the client.db breakup above, and likely as soon as I am done with IPFS–to create a new mappings cache layer that'll allow me to store and maintain accurate tag counts at all times. I expect it to slow down tag processing a bit, increase the total db size by a bit, and reduce autocomplete results generation in almost all cases to something like 250ms, possibly much quicker.
Of course, increasing the db size is something we absolutely don't want right now, so I'll be doing all this one step at a time. Expect the coming db updates to be pretty CPU and HDD heavy.
Once the db runs better, I will return to adding a suggested tag control and writing a faster dupe search algorithm.
full list
- the client's local server and local booru can be turned off from their respective management panels, and from now on, the client will initialise with them this way.
- if the local server or the local booru are not running, their copy/share commands won't appear in right-click menus
- the welcome dialog is now a simpler popup message
- incidence sorted tag lists are now sub-sorted by a-z lexicographic
- pasting many tags that have siblings to the manage tags dialog will ask you if you want to always preference the sibling, saving time
- added a 'clear deleted file records' button to the local file service on the review services window
- idle mode now cannot naturally engage within the first two minutes since client boot
- the autocomplete search logic will not count namespace characters in the autocomplete character threshold, so typing 'character:a' will not typically trigger a (very laggy) full search
- putting a '*' anywhere in an autocomplete search_text will force a full search, ignoring the a/c character threshold
- moved some specific 'give gui time to catch up' pause code to the generalised pause/cancel code that a lot of stuff uses, so big jobs should generally be a bit more polite
- split the daemon class into two–one for big jobs that remains polite, and another for small jobs that triggers regardless of what else is going on. this should increase responsivity for a number of scenarios
- fixed some bad wal failure detection and hence no-wal file creation on some instances of db cursor reinit (usually after service modification). because of now many superfluous no-wal files, existing no-wal files will be deleted on db update
- some external storage location errors are improved
- some internal and external storage location init is improved.
- if an error is detected in the external storage location manager, it will not attempt to rebalance again until the client is rebooted
- improved some upnp error catching
- cleaned up some misc shutdown thread-gui interaction error spam
- did some prep work on a future rewrite of daemon jobs pipeline
- split up some mixed file/data/404 'stuff was missing' exception code
next week
I'll try to finish off single-file IPFS upload, and I'll prep for this big db rewrite.
How would you people go about converting your whole GIF catalog into WEBMs?
Hey guys.
I'm going straight into it.
I've got 3.367 gif files above 5MB, amounting to 44GB. I can lower that to half or even less, just need a way to automate the tag transportation.
The fastest way I thought about was:
1) Export all gif files
2) Name all files with their respective tags
3) Batch convert everything to webm
4) Import everything back into Hydrus using a regex
I have everything sorted out except, you guessed it, 2). Just too many characters for Windows limits, therefore Hydrus won't export them.
The alternative is exporting all files with their hashes for file names and their tags inside txts, convert everything then manually add all tags back in!
I got to 24 files deep, before going insane.
Anyone has any ideas?
Release Tomorrow!
I had a good week. I mostly worked on the client's IPFS service, which now supports pin/unpin for individual files.
The release should be as normal tomorrow.
Feature request: APNG support
I have a fair amount of APNGs, but Hydrus only shows the first frame. Is it in the realm of possibility to get them working?
>pics related
user@ThinkPad-X60s ~/Downloads/hydrus-191 $ ./client.pyw
Critical error occured! Details written to crash.log!
user@ThinkPad-X60s ~/Downloads/hydrus-191 $ cat crash.log
Traceback (most recent call last):
File "./client.pyw", line 23, in <module>
from include import ClientController
File "/home/user/Downloads/hydrus-191/include/ClientController.py", line 1, in <module>
import ClientCaches
File "/home/user/Downloads/hydrus-191/include/ClientCaches.py", line 1, in <module>
import ClientDefaults
File "/home/user/Downloads/hydrus-191/include/ClientDefaults.py", line 2, in <module>
import ClientData
File "/home/user/Downloads/hydrus-191/include/ClientData.py", line 4, in <module>
import ClientFiles
File "/home/user/Downloads/hydrus-191/include/ClientFiles.py", line 7, in <module>
import HydrusFileHandling
File "/home/user/Downloads/hydrus-191/include/HydrusFileHandling.py", line 10, in <module>
import HydrusDocumentHandling
File "/home/user/Downloads/hydrus-191/include/HydrusDocumentHandling.py", line 2, in <module>
import PyPDF2
ImportError: No module named PyPDF2
user@ThinkPad-X60s ~/Downloads/hydrus-191 $ sudo pip install pypdf2
Downloading/unpacking pypdf2
Downloading PyPDF2-1.25.1.tar.gz (194kB): 194kB downloaded
Running setup.py (path:/tmp/pip_build_root/pypdf2/setup.py) egg_info for package pypdf2
Installing collected packages: pypdf2
Running setup.py install for pypdf2
Successfully installed pypdf2
Cleaning up...
user@ThinkPad-X60s ~/Downloads/hydrus-191 $ lather
user@ThinkPad-X60s ~/Downloads/hydrus-191 $ rinse
user@ThinkPad-X60s ~/Downloads/hydrus-191 $ repeat
Surely there must be a better way. Why can't you make a depends.py script? or is there one and i'm just a retard
Release Tomorrow!
I had a good week. I am still a little tired, but mostly back on schedule, and happy to be working again. I fixed a lot of bugs, improved some annoying gui things, and reworked some bad job logic to reduce various kinds of program lag. I also did some IPFS stuff, but due to a misunderstanding on my part on how IPFS works, had to cut it slightly short. I can still do some neat hydrus&ipfs stuff in future, but it'll take more work my end. I'll talk about it more in the release post.
The release should be as normal tomorrow.
hydrus dev
I noticed hydrus dev always posts classical art pictures when responding to threads. I imagine they probably have a big collection of it.
was hydrus network originally some sort of secret art cataloging/ art history analyzing program or do they just really like classic art?
Crashing since update
I updated to 191 and now it doesn't even open. I got this from the crash file though. How did I break it?
Traceback (most recent call last):
File "<string>", line 23, in <module>
File "c:\users\tom\appdata\local\temp\pip-build-l9fzm6\pyinstaller\PyInstaller\loader\pyimod03_importers.py", line 363, in load_module
File "C:\code\Hydrus\include\ClientController.py", line 3, in <module>
File "c:\users\tom\appdata\local\temp\pip-build-l9fzm6\pyinstaller\PyInstaller\loader\pyimod03_importers.py", line 363, in load_module
File "C:\code\Hydrus\include\ClientDaemons.py", line 14, in <module>
File "c:\users\tom\appdata\local\temp\pip-build-l9fzm6\pyinstaller\PyInstaller\loader\pyimod03_importers.py", line 363, in load_module
File "C:\code\Hydrus\include\HydrusServer.py", line 2, in <module>
File "c:\users\tom\appdata\local\temp\pip-build-l9fzm6\pyinstaller\PyInstaller\loader\pyimod03_importers.py", line 363, in load_module
File "C:\code\Hydrus\include\HydrusServerResources.py", line 13, in <module>
File "c:\users\tom\appdata\local\temp\pip-build-l9fzm6\pyinstaller\PyInstaller\loader\pyimod03_importers.py", line 363, in load_module
File "c:\python27\lib\site-packages\twisted\web\server.py", line 36, in <module>
File "c:\users\tom\appdata\local\temp\pip-build-l9fzm6\pyinstaller\PyInstaller\loader\pyimod03_importers.py", line 363, in load_module
File "c:\python27\lib\site-packages\twisted\web\util.py", line 304, in <module>
File "c:\python27\lib\site-packages\twisted\web\util.py", line 387, in FailureElement
File "c:\python27\lib\site-packages\twisted\web\template.py", line 401, in init
File "c:\python27\lib\site-packages\twisted\web\template.py", line 344, in _flatsaxParse
File "c:\python27\lib\xml\sax\init.py", line 93, in make_parser
SAXReaderNotAvailable: No parsers found
Hydrus newbie here, I've scanned the help section and haven't found an answer so I'll go ahead and ask here. My apologies if there's a general help thread; I'm somehow not seeing it in catalog view.
I've got a number of images that I've tagged via windows (.jpgs), or that I've written tags or other important details into the filenames (.pngs, .gifs, .webms, some .jpgs). My question is, is there a way to bring those tags into hydrus as I import? The ones that are in the filenames are easy enough, though time consuming, via manually typing them but the .jpgs with tags in their data I'm unable to see when importing.
Secondly, is there a way I could somehow force the filename to become a tag as I import? Not just for the files I already have named or tagged via filename correctly, but I like that I can look back over files I've picked up from the chans and know when I got them. Or does hydrus have some sort of "date created" tag that could read those in, instead of just the date when I imported them?
Importing subscriptions?
I see that I can export subscriptions in JSON format, but how do I import them?
No release tomorrow!
As expected, I spent the week traveling, so there will be no release tomorrow. Everything went great with my trip, and I am feeling a lot better after my previous illness. I now hope to be back to normal schedule for the foreseeable future, so the next release should be on the 17th February. Other than the normal bug fixes and small improvements, I plan to do a little more IPFS stuff.
Version 192
windows
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v192/Hydrus.Network.192.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v192.tar.gz
I had a good but very brief week. Unfortunately, it is taking me longer to recover from the flu, so I ended up sleeping most of the time. Otherwise, I have fixed a handful of bugs and added a neat subscription sync button.
I will be traveling next week, so there will be no release on the 10th. I expect v193 to come out on the 17th.
check subscription now
I have added a button to the manage subscriptions dialog, 'check on dialog ok', that will force the subscription to resync on ok, no matter when it would usually be due. Let me know if this doesn't work for you.
This has been on the to-do list for a while, but I realised it wasn't that difficult. I will add a similar one for repositories, so you can force repo sync without having to go help->debug->force idle.
misc stuff
Despite my illness, I did get my IRL and networking rebuild done. In the process, I updated my Win 7 development computer to a Win 10 machine. As far as I can tell, everything seems to be working, but in recreating my IDE and build scripts and so on, a help file might not be synced up to v192, or something github-related might be out of place. If you get that sort of error this week, please let me know!
The horrible Win 10 BSOD bug was fixed–it turned out to be bad Nvidia drivers! Version 361.43 was causing Photoshop and hydrus (and likely any other graphics-heavy program) to trigger a BSOD. Please update to 361.75, which came out about a week ago, and the problem should disappear.
I uploaded a new db init package this week, but I put it together incorrectly. If you downloaded the 1310 bare db that only had client.db in it, please get this corrected upload, which has all the client_… folders included:
If you do a lot of booru browsing and manual tagging, check out this thread, where a user has written a script to make it easier to manually copy tags from a booru into the manage tags dialog: >>1914
full list
- added a 'check on ok' button to the manage subscriptions dialog's subscription panel
- check file integrity now prints missing paths to the log
- fixed a typo that was breaking some repo sync download resumes
- if an external storage location does not exist on client boot, a popup will say so
- resetting a service is now much faster when the service is large
- some repo processing stuff is a bit faster
- improved some image rendering error handling
- some thumbnail errors are recovered from in a better way
- improved db update bit rot resilience
- improved the vacuum disk space error popup with better explanation of the problem and solution
- misc improvements
- moved to win 10 dev machine, so several windows libraries are a bit newer
- updated to opencv 3.1.0 on windows
next week
I will be traveling next week, so there will be no release. I will make a post on Tuesday the 9th to reiterate that, and then we should be completely back to a normal schedule. In the coming weeks, I would like to round out some more IPFS stuff and do a significant db rewrite, breaking off individual services into separate db files/folders that people can more easily import/export/delete/backup/share.
Crash on Startup
Just discovered hydrus so installed it but got a crash on startup. Here's the crash.log, help?
Traceback (most recent call last):
File "<string>", line 23, in <module>
File "c:\users\tom\appdata\local\temp\pip-build-l9fzm6\pyinstaller\PyInstaller\loader\pyimod03_importers.py", line 363, in load_module
File "C:\code\Hydrus\include\ClientController.py", line 1, in <module>
File "c:\users\tom\appdata\local\temp\pip-build-l9fzm6\pyinstaller\PyInstaller\loader\pyimod03_importers.py", line 363, in load_module
File "C:\code\Hydrus\include\ClientCaches.py", line 4, in <module>
File "c:\users\tom\appdata\local\temp\pip-build-l9fzm6\pyinstaller\PyInstaller\loader\pyimod03_importers.py", line 363, in load_module
File "C:\code\Hydrus\include\ClientRendering.py", line 2, in <module>
File "c:\users\tom\appdata\local\temp\pip-build-l9fzm6\pyinstaller\PyInstaller\loader\pyimod03_importers.py", line 363, in load_module
File "C:\code\Hydrus\include\ClientImageHandling.py", line 2, in <module>
File "c:\users\tom\appdata\local\temp\pip-build-l9fzm6\pyinstaller\PyInstaller\loader\pyimod03_importers.py", line 517, in load_module
ImportError: DLL load failed with error code -1073741795
Limited Release Tomorrow!
I had a good but very brief week. I mostly slept, still recovering from flu, but I found a bit of time to fix some bugs and add a 'check now' button to the manage subscriptions dialog. If I haven’t said ‘that’ll be fixed in v192′ to you and you don’t care about the subs button, then you can absolutely skip it!
The release should be as normal tomorrow.
I will be traveling next week, so there will be no subsequent release on the 10th–instead, v193 should be on the 17th.
I think this is pretty cool and I have a lot of pictures etc from Danbooru/Gelbooru (over 600GB) but I don't like the idea of having everything moved into one folder.
Is there no way to have it simply monitor my existing folders?
Safely closing Hydrus?
Is there a way to safely close Hydrus if it is in the middle of syncing with the public repository? I tried alt+f4, but it was still running in the background. When I force quit it and tried loading the client back up, the database was corrupted.
hydrus hanging
New user here, went all in with my stuff 2 nights ago and now having issues. can't access any files at all but database and files report as OK.Client has never closed cleanly since install. always hangs on "waiting for daemons to exit".
Any advice? Thanks in advance!
Retriggering Subscriptions
Is there any way I can trigger subscription downloads manually?
Version 191
windows
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v191/Hydrus.Network.191.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v191.tar.gz
I had a good week, but I was a bit short on time and my current dev situation is slightly up in the air. As well as some IRL stuff, I am currently upgrading to new computers and network and generally playing musical chairs with hard drives. I did however write a simple bridge to IPFS and fixed some bugs.
I will not be working much on hydrus next week as I finish moving my network stuff around and set up my new machines. As a result, I do not think there will be a release on the 27th Jan. I will be moving to a new Win 10 development machine, and assuming that all goes well, I will be looking into the Win 10 BSOD that recently appeared, so there might be a simple release that fixes that. Or, if there is anything critically wrong with today's release, I will put out a hotfix.
As a result, the next regular release should be on the 3rd Feb.
After that, I'm travelling for a week. I doubt I will have much online time, and I won't be able to dev at all, so there also will not be a release on the 10th.
Beyond that, everything should get back to normal.
ipfs
I have added a new IPFS service type, which you can add under services->manage services. You put in the IP and port of your IPFS daemon, and there's a little button to test the connection. Once you have that set up, you can go pages->new download page->an ipfs file, which will let you put in a multihash. While developing, I used QmUuNBmjSnDt191j3ZJTt1H3QQUa2mrRco2zf563RPYQeh, which is an image of a keyboard an anon posted in a /tech/ IPFS thread.
If you don't have IPFS and want to try this out, especially if you run Windows, I recommend you just go straight here:
https://gobuilder.me/github.com/ipfs/go-ipfs/cmd/ipfs?branch=release
I used:
https://gobuilder.me/get/github.com/ipfs/go-ipfs/cmd/ipfs/ipfs_release_windows-amd64.zip
Apparently the 0.3.x branch has been buggy for Windows, but it worked fine for me. It's a nice big ipfs.exe that you can just go–
ipfs daemon
–to and you are up and running.
The code on my end is rough and prototype. The error handling is bad, and the services->review services ipfs panel is placeholder, but I hope you can get it to work for one file or another. In future, I would like to add IPFS file upload and multihash->clipboard from a thumbnail's right-click menu so you should be able to effect very basic P2P between hydrus clients.
full list
- added ipfs service type
- added simple ipfs service gui to manage and review services windows
- added simple ipfs download and import to pages menu
- split confused service data object into a bunch of more flexible and readable subclasses
- raw url downloads are now powered by 'requests' rather than my hacked http engine, and work a hell of a lot better. more to follow
- raw url downloads now pause and cancel correctly
- optimised some a/c cache clearing in mapping update processing
- optimised content update object -> db yield logistics
- added no-daemons command line switch to disable all daemons
- added no-wal command line switch to disable WAL journalling for the db
- if the db fails to read a new WAL-journalled db file it has just created, it will delete the file and attempt to recreate the db, never trying WAL again
- changed the way the db writes big repo syncs to disk so that the job can be abandoned much more quickly
- improved local file not found error to state the directory or exact path that was the problem
- fixed some socket-level error reporting (I think timeout was part of this)
- the petition approval and denial client-to-server network commit is broken up into smaller steps to reduce server lag and the overall likelihood of timeouts
- subscriptions will no longer add files to their 'successful files' button when the file's url seemed new but it actually was previously deleted
- subscriptions will now identify themselves with an additional popup message when they encounter critical errors during sync
- editing a clientside service will force a reset of the appropriate session cache, so if you change access key, the account changes will be reflected immediately
- the import files dialog will no longer spam error popups on uninteresting or empty files–its 'parsing complete' statement is more verbose instead
- the 1.5s gap between parsing complete and the buttons being enabled is also removed. this was supposed to be helpful, to give you a moment to check the files were as expected, but in the end it was just annoying
- fixed a typo in some thumbnail error catching and regeneration
- improved my network code syncing, so linux and os x releases are cleared out of many old files
- misc code cleanup
next week
Again, there will probably not be anything more than a very simple bugfix release next week. I'll be mostly doing IRL stuff, and if I finish that early, I'll be taking a little time off. I will make a reminder and general update on what's going on post on the 26th.
No release tomorrow!
Unfortunately, I had a bad week. I went down with stomach flu on Thursday, so most of my hours disappeared. I eventually managed to catch up with my IRL stuff, and I rebuilt my network and dev computer as I planned, but I have had zero time for hydrus.
I now feel about 8/10 body and 9/10 mind, so I am keen to get back into it. Assuming I still have the energy, I hope to catch up on messages and reports over the next few days. If there is anything very important, please repeat it to my email or the v191 release thread. >>1855
I literally only got hydrus running on my new Win 10 machine in the past half hour, but neither the .py or the .exe have caused the BSOD, so there will be no easy fix there. I will have a think and make a plan.
So, there will be no release tomorrow. I expect to put out the next release on Feb 3rd. It will most likely be some bug fixes.
exhentai
How hard would it be to add exhentai to the gallery downloader?
tags
Hey hydrus dev, I noticed several items with tags for the artistic medium, do you have a guide of any sort on this tags to explain the difference between them and how to tell which is which?
Release Tomorrow!
I had a good week, but I did not have much time to work on hydrus. I had success with IPFS, including adding a very simple multihash->file downloads and imports to the client mechanism, and fixed several bugs.
The release should be as normal tomorrow.
As a general heads up: I will remain IRL busy for the next three or four weeks. There will likely be no/limited releases on the 27th Jan and 10th Feb.
Version 190
windows
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v190/Hydrus.Network.190.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v190.tar.gz
I had an ok week. Some IRL stuff reduced my hydrus time, but I otherwise figured out IPFS and fixed/improved some miscellaneous things.
IPFS
I downloaded IPFS, read through the docs, and played around with it to general success. I feel confident I can plug hydrus into it in a number of ways.
Next week, I will start coding. I will start with a panel to fill in your local IPFS daemon details and a test button, and the first actual action I would like to add is 'try to download this multihash from IPFS and import it to the client'.
win 10 bsod
This week, two users told me they were getting blue screens of death (full system crash) in Win 10 while running hydrus. This is obviously a serious error, and I would like to fix it ASAP. If you have experienced similar, please let me know your details and for how long you have been getting the problem. Please refer to these discussions:
http://8ch.net/hydrus/res/1714.html#1725
http://8ch.net/hydrus/res/173.html#1765
new file tables for the client
I have improved how the client stores file information and service->file mappings. A lot of file-related db stuff should be a little faster.
If you have a lot of files, your db might take thirty seconds or so to update.
full list
- fixed some hashing recalculation in collections that meant they reported no files in many cases
- fixed some hashing recalculation when 'remove files' is called
- improved the way the client db stores file information and service->file mappings
- idle processing jobs will now explicitly wake up as soon as the client naturally switches from not idle to idle
- the minimum allowed value for the 'max cpu %' control in the maintenance and processing options panel is now 5%
- the maintenance and processing panel is rewritten and laid out a little clearer
- 'busy' is now 'system busy' on the status bar
- force idle and force unbusy are now merged into a new 'force idle' that sticks until you explicitly turn it off
- busy and idle states should now update immidiately after closing the manage options dialog
- improved exit code event order to be less rude to the OS
- improved exit code emergency event handling
- fixed a typo that was not appropriately skipping the 'do you want to run shutdown jobs?' dialog
- file storage folder rebalancing will now occur on shutdown maintenance
- the client now closes down more reliably if the db fails to boot
- the client now closes down more reliably if the gui fails to boot
- if a client vacuum fails, it'll now also raise the exact error sqlite gave
- fixed ctrl+a on the autocomplete dropdown (I think the Insert/IME support change broke it)
- the ways the 'read' tag autocomplete control talks to other gui elements is improved
- the tag autocompletes will now refresh their results lists on more internal variable changes
- the query page management controller manages fewer redundant variables
- updated sqlite for windows
- the client and server dbs will attempt to change to TRUNCATE journal mode if WAL causes a read disk i/o error
- misc code cleanup
next week
I have a lot to do and not a huge amount of time right now. I would like to get some IPFS stuff done and deal with some serious bug reports like the Win 10 issue.
Crash on startup in v189
After a clean first-time install on Windows 8.1 x64, received the following error:
hydrus client started
booting controller...
booting db...
Traceback (most recent call last):
File "C:\code\Hydrus\include\ClientController.py", line 1033, in THREADBootEverything
File "C:\code\Hydrus\include\ClientController.py", line 442, in InitModel
File "C:\code\Hydrus\include\HydrusController.py", line 138, in InitModel
File "C:\code\Hydrus\include\ClientController.py", line 52, in _InitDB
File "C:\code\Hydrus\include\HydrusDB.py", line 44, in __init__
File "C:\code\Hydrus\include\HydrusDB.py", line 134, in _InitDB
OperationalError: disk I/O error
A serious error occured while trying to start the program. Its traceback will be shown next. It should have also been written to client.log.
shutting down controller...
hydrus client shut down
Afterward, the client was still running in the background, and had to be manually stopped from Task Manager.
I don't really know how to begin troubleshooting this, but running as an admin didn't make a difference.
has going through some shoddy setu
noticed two entires
I don't know what I'm doing >
I was lazy and didnt update for almost a year.
What is the best way to upgrade from version 152 to 190? Just install 190 over?
Version 189
windows
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v189/Hydrus.Network.189.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v189.tar.gz
I had a pretty good week. I didn't get around to everything I wanted, but I made significant improvements to some formerly big and slow gui and db stuff.
some slow things are faster
I have split the analyze call that usually happens on db update into a lot of smaller jobs that'll be tucked in with the other idle maintenance routines. I shouldn't expect you will ever notice it again. This means that the v188->v189 update step should only take about one second!
I have also improved how the db prepares for a vacuum. It seems to go a little faster now, and if you run Windows, your next vacuum will change something important inside the db that should reduce first-load hdd latency. Let me know if you think your Windows system is running searches faster in the coming week!
Also, I sped up the gui code that adds new thumbnails to an existing set of results, like for file import pages. It was running fast with a small number of thumbnails, but not when there were many thousands. I figured out what was taking so long and rewrote it, and for an example page of 5,000 richly tagged results, I managed to cut ~266ms of processing time down to 3ms! I hope this speeds up some people's large imports in future.
Server and client backup code will also work a hell of a lot faster if you backup onto an existing backup–now, files that have the same size and 'last modified' date (which includes all your regular files and thumbnails, which are not changed by the client) will now be skipped rather than overwritten, saving a lot of time.
Also, I rejigged some maintenance stuff so the client should be able to more quickly respond to a change from idle to not idle when it is in the midst of something big.
full list
- split the big analyze db calls into individual table/index calls and moved them from update code to the normal maintenance routines
- on vacuum, both the client and server dbs will now bump their page size up to 4096 if they are running on windows (server vacuum is triggered by running a backup)
- vacuum should be a slightly faster operation for both the client and server
- boosted the db cache significantly–we'll see if it makes much difference for things.
- the way the selection tags control updates its taglist on increases to its media cache is massively sped up. An update on a 5,000-thumbnail-strong page now typically works in 3ms instead of ~250ms. large import pages should stream new results much more quickly now
- sped up some slow hash calculation code that was lagging a variety of large operations
- some hash caching responsibility is moved about to make it available for the add_media_results comparison, which now typically works in sub-millisecond time (was about 16ms before)
- some sorted media list index recalculation now works faster
- some internal media object hashing is now cached, so sorted list index regeneration is a bit faster
- some medialist file counting is now superfast
- wrote a new pauser object to break big jobs up more conveniently and reduce gui choking
- the repo processing db call now uses this pauser
- some copy and mirror directory functions now use this pauser
- backup and restore code for the client now skips re-copying files if they share the same last modified date and file size
- backup code for the server now skips re-copying files if they share the same last modified date and file size
- http cannotsendrequest and badstatusline errors will now provoke two reattempts before being raised
- socket error 10013 (no access permission, usually due to a firewall) is caught and a nicer error produced
- socket error 10054 (remote host reset connection) is caught, and the connection is reattempted twice before being raised
- the old giphy API is gone, so I have removed giphy
- forced shutdown due to system exit/logoff is handled better
- pubsub-related shutdown exceptions are now caught and silenced
- an unusual shutdown exception is now caught
- fixed a copy subtag menu typo
- cleaned some misc hydrus path code
- tags that begin with a colon (like ':)' ) will now render correctly in the media canvas background
- some misc code cleanup
- dropped flvlib since ffmpeg parses flv metadata better anyway
next week
I still have some things in my github queue, and my private to-do list remains stuffed, but I feel I have improved a lot of broken and slow things over the past month or so and would like to push ahead with something big and new. Thank you all for voting in the poll I set up:
The top results right now are:
- IPFS plugin
- suggested tags control
- faster dupe searching
So I will have a bit of a think and get going on those. I don't know how long each will take, but I think I will put about a third of my time into this stuff and see how that works out.
I will start with IPFS , although I still do not know a huge amount about it. I am fairly certain I can upload files into the network, but searching and browsing and hence having a hash to download is something I can't confidently state. Anyway, I will try to get a very simple bridge between hydrus and IPFS working, just to see if it all works.
Otherwise, I will keep working bugs and cleanup.
when
quick suggestion: filter shortcuts
Any possibility that the "filter" -> "archive/delete" / "custom filter" submenu could have saved filters added to it? Would save me one click when remoting, which would be awesome in that sort of "whoa minor efficiency upgrade" kind of way.
Release Tomorrow!
I had an ok week. I played with IPFS and had success, fixed a couple of bugs, and improved existing code.
The release should be as normal tomorrow.
Crash on startup
Hi there. I updated from 181 to 189. Now client.exe crashes on start, leaving behind a crash.log. Not sure if I missed something? I downloaded the zip and extracted on top of the existing installation, as I did for every update.
Traceback (most recent call last):
File "<string>", line 23, in <module>
File "c:\users\tom\appdata\local\temp\pip-build-l9fzm6\pyinstaller\PyInstaller\loader\pyimod03_importers.py", line 363, in load_module
File "C:\code\Hydrus\include\ClientController.py", line 3, in <module>
File "c:\users\tom\appdata\local\temp\pip-build-l9fzm6\pyinstaller\PyInstaller\loader\pyimod03_importers.py", line 363, in load_module
File "C:\code\Hydrus\include\ClientDaemons.py", line 14, in <module>
File "c:\users\tom\appdata\local\temp\pip-build-l9fzm6\pyinstaller\PyInstaller\loader\pyimod03_importers.py", line 363, in load_module
File "C:\code\Hydrus\include\HydrusServer.py", line 2, in <module>
File "c:\users\tom\appdata\local\temp\pip-build-l9fzm6\pyinstaller\PyInstaller\loader\pyimod03_importers.py", line 363, in load_module
File "C:\code\Hydrus\include\HydrusServerResources.py", line 13, in <module>
File "c:\users\tom\appdata\local\temp\pip-build-l9fzm6\pyinstaller\PyInstaller\loader\pyimod03_importers.py", line 363, in load_module
File "c:\python27\lib\site-packages\twisted\web\server.py", line 36, in <module>
File "c:\users\tom\appdata\local\temp\pip-build-l9fzm6\pyinstaller\PyInstaller\loader\pyimod03_importers.py", line 363, in load_module
File "c:\python27\lib\site-packages\twisted\web\util.py", line 304, in <module>
File "c:\python27\lib\site-packages\twisted\web\util.py", line 387, in FailureElement
File "c:\python27\lib\site-packages\twisted\web\template.py", line 401, in init
File "c:\python27\lib\site-packages\twisted\web\template.py", line 344, in _flatsaxParse
File "c:\python27\lib\xml\sax\init.py", line 93, in make_parser
SAXReaderNotAvailable: No parsers found
Deviantart scraper not getting full size images
Every gallery preview image links to a medium size image with a download link in the upper right. Hydrus is downloading the medium size image where it needs to be following the download link.
Hydrus Speed
Hello again. I've been using Hydrus for a few days. Well, rather I've used it before and I'm getting back into it. I wanted to organize all the things I've been pulling from pixiv and tag them based on danbooru tags. That way I could export and appropriately dump art.My main problem right now is the speed, it was the same problem when I first tried using hydrus months back. At first, it's really fast and could grab my 3000 images for me with no problem, but after I started adding more tags, it got much harder. I used the danbooru tag archive you had on mediafire and now it's just outrageously slow.
Right now, I'm having it search for all the files I have under series:touhou. I started the search at 14:01 EST. It's currently 14:14 as of this sentence and it has made it to "768 of 1566". So, I'm looking at 256 images every ~4 minutes. I don't know if this is a normal thing, if I failed to set something up, or if my 8 year old prebuilt Gateway desktop that I'm running this on is just not cut out for mass image tagging. But I'd appreciate any kind of information of reasons as to why this isn't working out.
Release Tomorrow!
I had a good week overall. I fixed quite a few bugs, improved some network error recovery, and rewrote some slow gui and database code so that many large jobs will run much faster. Importing into a page that already has many thousands of files is particularly faster.
The release should be as normal tomorrow.
Deviantart tag scraping isn't working today
I added several subscriptions to DA last night and most of them pulled the artist and title tags from the site just fine. I had to shut down the computer with about 500 images still in the queue. When I started Hydrus just now it started importing the images but no tags.
Any ideas? I thought I might need to log in to the site in a browser to get some kind of IP-based cookie, but it hasn't worked.
Version 188
windows
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v188/Hydrus.Network.188.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v188.tar.gz
I had a good week. I took some time off and then fixed some bugs and worked on some rewrites. Most of the changes are small improvements–you might notice some text updating faster or your gui lagging less during large operations.
If you missed it, I created a new poll last week for the next thing to focus on:
If you have an opinion, please do vote. On the 6th of January, I will check for the top two or three things and start spending a bit of time every week working on them in the background. It looks like an IPFS plugin is currently winning.
Also, the Anon who produced some tag archives has made some more at >>1668 . I improved how the client syncs to tag archives this week (the initial sync is pausable/cancellable and won't kill the db/gui so much), so if you use this stuff a lot, let me know how you get on.
more hash searching options
system:hash now also supports md5, sha1 or sha512. The dialog where you enter its value now has a dropdown to select which hash type you want. These new ones work just like sha256–put your hash in in hex, and if it exists locally, it should appear. These additional hashes are only known for local files, so this will not work for remote files.
system:hash is a rarely used predicate, so I didn't bother to write code to update any existing stored system:hash predicates to the new system. If you do happen to have a v187 session that stores one, they will break in v188. Please replace those search pages with 'open selection in new page' pages or whatever before you update.
full list
- if you have custom filters set up, they will now be listed in the normal thumbnail right-click menu, which will expand for them
- 'refresh account' button's event no longer waits for service response on the gui thread
- 'refresh account' button on review services will now re-enable if the call fails
- added 'hide inbox and archive predicates' checkbox to options->default file system predicates. not sure if this is the best place for it!
- system:hash can now query md5, sha1, and sha256
- existing pages with system:hash will break on update. since this is such a rare predicate, I assume this is not a problem!
- improved db hash cross-reference flexibility
- any pausable popups will now explicitly change their text to 'paused' when they are paused
- upload pending popup now has a title, so it should pause a little more sanely
- regenerate thumbnails popup now has a title, so it should pause a little more sanely
- cleaned up some job pause status logic
- upload pending now works on a stream, so gigantic uploads will cause fewer hitches and overall problems
- upload pending code is cleaned and simplified in several ways, with better error handling and progress reporting
- synctotagarchive is now split into a thousand tiny separate jobs and runs on a separate non-db thread. it will no longer lock the db the whole time
- synctotagarchive popup should be a lot more responsive and will specifically give the gui some time to catch up after every chunk of data processed
- synctotagarchive will update pending tags count (and gui stuff will show the pending tags) as it goes
- synctotagarchive is now a pausable and cancellable operation!
- large repository sync jobs should be a little less laggy on final commit
- system:duration now displays its value as a properly formatted milliseconds time delta, rather than just a big int
- profiles now flush to the log immediately
- some cpu-heavy daemon recheck periods have been extended to reduce potential hangs on slower computers
- some repository sync messaging wording and logic is improved (it should report what it is doing more reliably when you are cpu-busy)
- reordered instances of 'waiting politely', so download text/gauge updates should occur before the wait does
- some misc 'waiting politely' logic is improved
- pixiv ugoira is detected and a better error (that will be ignored by the new sub error throttling code, for instance) is produced
- 'uninteresting mime' errors have nicer import status note text
- fixed the 'thumbnail grid background' custom colour option
next week
I took some time off my github queue this week, so I will jump back to that. I also want to work on speeding up some newly identified slow code, and maybe reorganising some maintenance routines so they run in a neater way. I want to ditch the analysis run after every update.
Tagsistant
Hey hydrusdev, have a look at Tagsistant sometime. It's a FUSE filesystem that you mount and create folders to represent tags, and then you make queries out of the paths into the filesystem.
Might give you some ideas for your syntax and other interfaces.
Reposting from >>>/tech/457184
Maybe relevant to your interest.
Japan did it.
Automatic anime images tagging with neural networks.
THE FUTURE IS NOW
http://illustration2vec.net/papers/illustration2vec-main.pdf
I had a good week. I fixed some bugs, rewrote some laggy big jobs, and added some small new things. I got more done than I expected.
The release should be as normal tomorrow.
Version 187
windows
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v187/Hydrus.Network.187.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v187.tar.gz
Merry Christmas!
I had an ok week. I was a bit busy with IRL end-of-year stuff, but I caught up with more reported bugs. There is nothing to particularly highlight this week–I think everything is just little fixes.
I am slowly churning through my github bug reports, and I am on track to be somewhat 'done' with the backlog in the next week or two. As a result, here is the long-awaited poll on what to next prioritise:
Check it out!
On the 6th of January, I will select the most popular two or three things and start work on them in order. I am very interested to see the results. Feel free to discuss the items here or elsewhere, and if you have any questions, let me know.
As a small reminder, if you are a Windows user and you are still running v185 or before, there are special update instructions because of the 32-bit to 64-bit change in v186 and beyond. Please check out the v186 release post:
http://8ch.net/hydrus/res/1589.html
And in other news, a user posted several neat new tag archives here:
http://8ch.net/hydrus/res/290.html#1619
full list
- file->restart now works for linux
- the 'call to' thread pool now has a delayed start, which fixes a shutdown hang in linux (and perhaps the other platforms, in one way or another), when the user hits 'forget it' on the 'already running' boot dialog
- removed some console error junk on abandoned client boot
- if a requested full-size thumbnail is missing, the client will now attempt to regenerate it
- if a requested full-size thumbnail does not render (for instance, during a request to generate a resized thumbnail), the client will now attempt to regenerate it
- some thumbnail fetching and generation logic is slightly improved
- you can now drag animated media around when the inital drag click begins over the animation canvas
- ditched some ultimately unhelpful mouse warping when the cursor was near the edge of the media viewer when a drag began
- possibly fixed a strange tags_fts4 mirroring problem that has hit some users for a reason I do not understand
- import tag options panels will no longer order their tag services randomly
- fixed a bug where import folders would not save changes to their explicit tags
- fixed a bug where under some circumstances, the gallery importer was attempting to display previously deleted files on the screen, causing errors.
- subscriptions will now pause a few seconds after a file import error, which should solve a problem with client hanging when lots and lots of subscription file errors occur one after another
- if subscriptions encounter 5 file import errors, they will pause their sync and try continuing later
- pixiv manga error handling is improved
- bad mime error handling is improved for all importers
- fixed an important bug in the import status cache where error texts were not being stored as unicode, so error texts with non-unicode-compatible characters were causing exceptions when the session was being saved to the db or the import status frame was being loaded
- existing sessions' import status caches should automatically convert their 'notes' to unicode on update
- removed booru db fetch spam when booru download page has no work to do
- fixed a typo in regenerate thumbnails popup message
- fixed some bad event handling for the gallery page's file limit control
- updated some regex defaults to have better multiplatform support and handle a wider array of valid characters
next week
I expect I will be a little short on time again, so probably another week like this.
Fuuuug
Also we really should improve the tag parents/siblings ui
probably add filters and shit
becomes a dear mess when it's full of scattered entries
Having trouble with two things, not sure if i should be posting here or not for help.
First up Is there a quicker way to scan for duplicates of all images rather than individually?
Secondly is there a way to stream line translate tags from Pixiv to danbooru tags at all?
Version 186
windows
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v186/Hydrus.Network.186.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v186.tar.gz
I had a good week. I did some small things and moved the Windows build to 64-bit, which turned out to be a more significant change than I expected. If you use the Windows release, you will need to perform a clean install this week. Explanation below.
Windows 64-bit
Converting my Windows dev environment from 32 to 64-bit went fairly well. Almost everything compiled ok, and I was able to source builds for the other things that caused trouble.
In the client, the old 32-bit ~2GB memory limit is of course gone, and things are surprisingly snappier. Webms now decode much more smoothly, even at 1080p. I don't think I am imagining the speed improvement–I assume my laptop has 64-bit media decoding that the new 64-bit ffmpeg exe can hook into, and I presume Windows prefers to display things on screen from a 64-bit exe.
However, when I just tried to update my laptop by dragging and dropping the 64-bit install over the 32-bit, it wouldn't boot. It looks like some spare old 32-bit dlls are conflicting with the new 64-bit exe. You will have to perform a clean install.
performing a clean install
First, if you run a backup every week, do it!
Second, go to your install directory. If you don't know what this is, go file->open->installation directory in your 32-bit client and then shut the client down.
Delete everything in that directory except the db folder.
Then update to the new version as normal, whether with the exe installer or by extracting from the zip.
You have now performed a clean install! The new 64-bit client/server will pick up exactly where it left off with your old database. I think that process is simple, but if you are still uncertain, don't be afraid to ask for help. Do not delete your install_dir/db folder.
misc other stuff
I wrote a new help page for reducing program lag, which generally summarises several conversations I have recently had with users–check it out if your client is running slow.
https://hydrusnetwork.github.io/hydrus/help/reducing_lag.html
I also changed last week's new autocomplete shortcuts due to a good comment about them breaking tab navigation. IME mode now uses Insert, and 'get a/c results now' now uses Ctrl+Space. I will make these editable, hopefully along with all other shortcuts, when I get around to rewriting the shortcut system.
I added file->restart for Windows and OS X. I will fix it for Linux next week.
full list
- Windows release is now 64-bit!
- some libraries got updated for windows, to varying minor effects
- BeautifulSoup updated to latest version for everything
- fixed a warning from newer versions of BeautifulSoup by explicitly using lxml
- updated the running_from_source help file to what is currently needed
- ffmpeg.exe is also 64-bit
- wrote a "reducing program lag" page for the help
- added program restart to file menu, doesn't work for linux yet
- improved restart code (it was used in restore database, and for some users would not close the parent process down properly)
- in the media viewer, page up/down, arrow keys and mouse scroll no longer unhide the mouse
- in the media viewer under Windows, dragging no longer unhides the mouse
- the implicit delay in the downloader-importers is reduced, so redundant files should always stream in a super fast now
- export files dialog will now auto-create missing destination directories
- export files dialog will now work off the gui thread (to stop gui hangups on big jobs) and will report its export status through the export button, which will be disabled until the job is finished
- changed the new tab/shift+tab shortcuts because they break panel control navigation (I want to make these editable soon):
- 'fetch autocomplete results now' shortcut is now Ctrl+Space (Raw Control for OS X, not command)
- 'IME mode toggle' is now Insert
- cleaned up how control is handled cross-platform–in general, 'control' is always control for windows and linux, and 'control' is command for os x. this may be incorrect.
- json dump errors now report more information about the specific object and its serialisable contents that are causing the problem
- clipped sibling/parent reason entry dialog message when lots of pairs are added
- fixed a bit of bad logic in sibling/parent dialog->content-update workflow
- fixed the thumbnail canvas not refreshing on a canvas shrink caused by expansion of the management panel that doesn't change the thumb column count
- fixed emergency boot error reporting for when HydrusData won't load
- added some unicode conversion unit tests
next week
I made some more progress on my github queue this week. I think I will be able to work on something new in a couple of weeks. I plan to put up a large voting list in next week's release post so people can vote on what they would like me to start on the week after. Here is my current rough list of ideas, made from specific suggestions and my own mental list of things I'd like to get around to but don't usually have time for:
- improved censorship controls, hiding namespaces from certain views and so on
- suggest tags control ("people who tagged like that also tagged like this" workflow)
- tagging filter workflow (like inbox/archive)
- dupe search
- display search results in different ways (like a file explorer)
- support text and html files
- audio support
- ipfs
- optimise search queries (make numtags faster, for instance)
- have search results stream in rather than loading all at once once the whole query is done
- local booru improvements
- gallery downloader improvements
- client-to-client communications, im or email
- better parents and siblings dialogs
- streaming (much faster) a/c results
- better shortcuts
- better undo
- remote ratings
- tiling media canvas (big images, better zoom quality)
- tag cloud control
- mass adding/editing of subs
I will polish the phrasing before the poll to more specifically explain each item, but I hope that if you are an avid user you will generally know from my previous comments what those things generally are. If not, feel free to ask!
In the meantime, please think about which of those things you would most like to see. I would enjoy working on all of them.
If there is something you care about more than any of those things that I have forgotten, let me know and I'll add it.
Release Tomorrow!
I had an ok week. I was a bit busy with the holidays coming up, but I fixed quite a few bugs, mostly to do with gallery downloads and subscriptions.
I will put a vote up on tomorrow's release post for what to work on next. Since many of you are equally busy with end-of-year, I will leave it up for two weeks and aim to start work on something new on the 6th January.
Public Test Repo Tags slow down client
Syncing to the PTR Tags only causes searching and even tagging to become unbearably slow. Tagging will take roughly 15~ seconds to open, which is simply unusable.
Am I doing something wrong? Or is this simply a problem?
client is laggy and slow
hi there,
i've been using hydrus for a while now and am pretty happy with it. the only inconvenience i'm experiencing is the program being slow to respond from time to time.
most often this happens when i type the first half of a tag and the program begins crawling tag suggestions. then the program locks up for about a minute and then works fine again.
my db is about 7 GB in size and i run the program from a HDD partition that is separate from my OS partition. i run windows 7 64bit and my hydrus is version 181 (will update to 185 now).
from reading how other people here have been importing *booru images en masse, i'm getting the feeling that my 8k images are still relatively few.
is there anything i can do to make hydrus operate more smoothly or is my db too large?
Release Tomorrow!
I had a good week. I fixed some small things, improved some small things, and moved the Windows release to 64-bit.
The release should be as normal tomorrow. I'll talk a little about the next thing to work on, as well.
Version 185
win32
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v185/Hydrus.Network.185.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v185.tar.gz
I had a good week getting a lot of miscellaneous stuff done. I cut my github queue down significantly. I think I might be free to start work on something big and new in a couple of weeks.
I belatedly updated the sqlite dll for Windows this week. If your v180-ish Windows server does not boot, please update to v184 first, which will remove an index from the db that is incompatible with the new dll!
discard/require/permit/exclude
Right-clicking on some 'selection' tags or on the active 'search' predicates will give you some new menu options to add or remove inclusive or exclusive tags, so it now only takes a couple of clicks to do something like excluding a tag from the existing search. Just try right-clicking on some selected files' tags and have a play with it. The logic behind some predicate comparison and moving about is improved as well, so the workflow in general of this should be a bit less janky.
I didn't like the idea of spamming 'add x/remove x/add -x/remove -x', so I went with richer verbs. Let me know what you think.
:)
Unnamespaced tags that start with colon should now be entirely supported! At the db, gui and network stages, either on the client or the server, these tags should keep their leading colon (previously, they would lose it, so ':<' became '<'. I can't retroactively fix the broken tags, but adding siblings to manually fix them should work, and future booru/whatever parsing should work.
Obviously if you discover a situation where these tags still break, let me know!
misc
Subscriptions, import folders, export folders, and the thread watcher now support finer control of their checking periods, so where before you might have seen days, you now see days and hours. If you use any of these, just check them out and you'll see.
The main gui's menus are streamlined and grouped slightly more sanely.
There are two new options: in 'speed and memory', you can add a forced system:limit that will apply to all queries, and in 'downloading', you can choose to replace the new traffic light control with some text.
Subscriptions now report their number of failed files, if they have any, on their dialog panel. Buttons to quickly go 'retry all failed files' and 'retry all failed for all subs', and perhaps some other commands, are planned for the future.
full list
- right clicking on a tag/predicate in the 'selection tags' box or the 'search' list of active predicates provides new intelligent menu options to discard/require/permit/exclude the selected tags from the current search. try it out!
- this system also works for namespace and wildcard and system predicates
- 'predicate a' and its inverse '-predicate a' are no longer considered the same!
- namespace and wildcard predicates will now remove their inverts if they are added to the list of active predicates
- the way inverse predicates are generated and compared is generalised and improved
- fixed 'open a new search page for tag x', which might have been broken?
- quick-entering a tag in the 'read' tag autocomplete entry will now always replace with a sibling if one exists
- quick-entering a tag in the 'write' tag autocomplete entry will now not replace with a sibling–you will get the option of what you want to do
- when this happens, the preferred sibling is labelled on the 'what do you want to do?' popup in the manage tags dialog
- also when this happens, the affected file count for add/pend the preferred sibling should now be correct
- fixed client support for tags that begin with ':', such as many emoticons. these should now be parsed correctly from websites and keyboard input, keep their leading colon through network and db conversions, and display correctly (for the most part!), despite a little bit of magic to make them work
- created a new TimeDelta control to handle finer tuning of optional periods
- improved timedelta->pretty string code to support >60s times
- manage subscriptions dialog now supports day and hour period, with min period of four hours
- manage export folders dialog now supports day/hour/minute period, and the parent dialog displays that time delta appropriately
- manage import folders dialog now supports day/hour/minute period, and the parent dialog displays that time delta appropriately
- the thread watcher now supports hour/minute/seconds check period
- folded 'search' and 'download' menus into 'view' and renamed it 'pages'
- 'sessions' moved from 'file' to the new 'pages'
- moved 'admin' menu to 'services->administrate services'
- added 'forced system:limit' to options->speed and memory panel
- if several system:limits exist in a search, the minimum will be used
- added a checkbox to options->downloading to replace the traffic light waiting politely indicator with text
- subscriptions with failed files will say 'x urls in cache, y failed' on their dialog panel
- at the gui level, media that superfluously receives a 'pend tag' instruction for a tag that is already current will discard the instruction (this improves the accuracy of the pending tag count during and after the manage tag dialog)
- and the same thing for 'petition tag' when the tag isn't already current
- any accidentally added 'namespace:'-type tags will be deleted on update
- fixed a bad merge in the manage tag siblings dialog's autocomplete dropdown lists
- the thread watcher can now deal with urls with a #54951495 post anchor at the end
- refactored some controller manager code
- removed some useless old code
- cleaned some misc code
- improved some bad old orphan deletion code
- deleted some old code
- finished off some old media result streaming code
- refactored a bunch of search stuff from clientdata to clientsearch
- rewrote some subs gui/help text to be a bit clearer, and added a help link to the dialog
next week
I want to move to 64-bit python for Windows and hence be releasing 64-bit builds for all platforms. Updating my Windows dev environment might be easy or difficult, so it will be the first thing I work on, and if it all goes wrong, I will reset and have a rethink. After that, I have 53 cleanup jobs, which I will triage, and more github stuff to get to.
Large Feature Request: Do I already have this image feature
I have a feature request that I think might be fairly large. I could be wrong
Can we get a feature that allows someone to input an md5 hash into the program, and it scans your collection and tells you if its already in hydrus, perhaps make it runnable standalone along with hydrus, as to make it a little more lightweight.
I run into a LOT of images, that I have no idea if i have or not, and I want to know if i already have the image. If I could grab the md5 and have it scan my hydrus library, i could cut down on my duplicate file saving a lot.
I am considering this as its own program, with its own database that gets generated, but im a novice, so it would take a long ass time to learn how to code something like that, and it might be more useful if it were something already integrated with hydrus
thoughts?
Version 184
win32
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v184/Hydrus.Network.184.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v184.tar.gz
Despite being ill the whole time, I had a great week. I have completely cleared out my old bug to-do list and I added support for external file storage.
file storage location(s)
In the new options->file storage locations panel, you can now designate additional folders for your client to use to store files. You give each location a weight, and the client will incrementally balance all your files proportionally across them. Once you have a desired balance set, the client will move small amounts at a time in the background, or you can force an immediate full rebalance from database->maintenance->rebalance file storage. I hurried this system to solve two problems:
1. Your collection is larger than any single hard drive you own.
2. You want to run client.db from a small fast drive like an SSD but would prefer to keep your read-only files on a larger slower HDD.
This is still in a prototype stage, so please do not play with it if you are a new user or do not maintain a backup.
I have attempted to make this system flexible and to fail gracefully. If you click on a thumbnail and the file or location responsible for that file is missing (e.g. you unplugged the usb hard drive they were on), you will get some popup errors but nothing terrible will happen. If the system discovers subfolders in the wrong location (e.g. your drive paths changed and you are trying to put everything back together), it will silently attempt to heal the damage by putting things where they are supposed to go. You can remove the original install_dir/db/client_files from the list if you like (as long as you've added something else to take over), and that folder will slowly lose responsiblity for storing anything at all.
If you use this, let me know how you get on! I'm obviously interested in any bugs you encounter. I expect I will write some formal help for it, so let me know whatever confused you. Some common-sense rules apply for this initial release–don't try to share the same external storage location between different clients, and don't force a rebalance while watching your webms, as the system won't be able to move open files.
maintenance and processing
If you have special maintenance and processing idle-timing options set, please recheck them, as that panel has had a minor makeover! Figuring out when idle time will kick in should be a bit more intuitive now!
autocomplete improvements
I have tweaked the low-latency optimisations I added to the autocomplete entry last week, and also added a couple of shortcuts:
Tab – This will now force the autocomplete to go fetch autocomplete results immediately. A new option in options->speed and memory also lets you completely disable automatic result fetching, so use Tab if you want to only fetch results manually.
Shift+Tab – This will disable/enable capture of Up/Down/Enter key events so you can enter non-english text with an IME without having your IME key events stolen for the dropdown list.
waiting politely indicator
Whenever a downloader waits for a few seconds, instead of spamming 'waiting politely' status text, it will now turn a little 'traffic light' icon from green to yellow. If you do a lot of downloading, let me know what you think of this! I expect I could do more with it. Turning red on an error, perhaps.
full list
- added external client_files storage!
- you can add external client_files folders in options->file storage locations, further giving them weight
- a new daemon will incrementally rebalance your files (and recover orphaned subfolders!) over your different storage locations
- you can also force a full rebalance from the new database->maintenance->rebalance file storage
- simplified the maintenance and processing panel
- the maintenance and processing panel controls now appropriately support being set to 'none'
- the 'run jobs on idle?' question is now explicit on the maintenance and processing panel
- deselecting 'run jobs on idle/shutdown' will now disable subordinate controls
- hitting tab on the autocomplete control now triggers an immediate 'fetch results' call
- added a checkbox to options->speed and memory to completely disable automatic autocomplete results fetching (i.e. if you want to manually control a/c result-fetching only with this new tab shortcut)
- the new less-laggy autocomplete results fetch won't trigger if the latest query is shorter than the cached query (i.e. you won't get lag when hitting backspace a bunch of times on autocomplete)
- hitting shift+tab on the autocomplete control now disables all other key event capture, letting you enter IME without your up/down/enter presses controlling the dropdown list of results
- 'waiting politely' time is now indicated with a small 'traffic light'-type circle control on all downloader pages
- fixed some occasional 'I'll go sit in the top-left corner of the screen and not fix my position' hover window behaviour in Linux
- fixed sibling predicate collapse for the 'read' autocomplete dropdown for database results
- refactored how predicates are collapsed and sorted to be a bit more sensible
- fixed maintenance of local booru data use even if local booru does not receive any requests
- improved some service/content update error handling in repository sync
- the system:rating value-entry dialog no longer lists its rating services in random order
- 'open externally' will no longer show for non-local thumbnails' right-click menus
- fixed audio/pdf thumbnail display for local booru, although they will be full size for now
- harmonised some hex-prefix folder generation code
- improved some file and directory copying code
- improved upload pending popup message cleanup
- added help_dir to hydrusconstants
- miscellaneous refactoring
- misc cleanup
- extracted server-specific services and server resources from the common import path to the server import path
- a bit more misc server refactoring
- updated a couple of bits of help
next week
My local to-do list is now free of bugs, and I don't think I have anything new to add that is too urgent, so I think I can focus more on my github queue. So, I'll work on that, and do some code cleaning and add some more unit tests.
Release Tomorrow!
I had a good week. I fixed some bugs, improved some gui controls and tag handling, and added some new options.
The release should be as normal tomorrow.
Release Tomorrow!
I had a great week. I was ill the whole time, but I fixed a lot of old and new bugs and added multiple hard drive support (i.e. you can store your files across multiple external locations now).
The release should be out roughly as normal tomorrow.
Version 183
win32
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v183/Hydrus.Network.183.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v183.tar.gz
I had a great week. My new schedule worked out very well. I cleared out many of my oldest to-do items and added thumbnail support for swf files.
swf (flash file) thumbnails
Swfs now have thumbnails! The system works ok–many files have artifacts, and some only have blank rectangles, but most show something to do with the movie or game. It is a huge improvement over just showing generic filetype thumbnails.
The client and server will generate thumbnails for existing files on update to v183. It worked at about 2 swf files/second for me, so if you have 5000 swfs in your client, be prepared to wait a little while. I got a handful of os-level 'swfrender_win32.exe just crashed' popup windows during my big update, but they didn't seem to affect the client at all–from the client's perspective, if swfrender has any problems, it just defaults to the old filetype thumbnail.
This is all a bit experimental, though, so if you run into serious problems, let me know!
Clients will attempt to download thumbnails for swf files from file repositories, so if you have a server that has some swfs, make sure you update this week or you'll have a few inconvenient 404s flying around. I thought about updating the network version to force everyone to update, but I figured that would ultimately be more of an inconvenience.
misc stuff
I have fixed a lot of old, weird, and boring issues this week. Feel free to read through the full list if you have been waiting for something specific, or just update and move on–there isn't anything really important, but a lot of things are generally a bit better.
I made some progress on my github queue as well, this week. I hope to cut it down a bit every week until I hit inbox zero.
full list
- added swf thumbnail support–it works ok for most swfs!
- thumbs for existing swf files will generate on db update for both client and server
- the server will also generate thumbnails for video on update, which it seems it was not doing before
- rewrote some hash comparisons in thumbnail downloading and thumbnail counting code to be a lot faster and memory efficient
- fixed thumbnail count invalidation after service sync
- in certain cases, fetching autocomplete tag results from the db will be slightly less laggy
- if the autocomplete dropdown has cached or otherwise quickly fetchable results, it will refilter for current input as you type, ignoring the normal character delay (i.e. typing through autocomplete is far less laggy now)
- fixed a couple of autocomplete tag-parsing bugs for tags with more than one colon character
- fixed some key event selection bugs in the autocomplete taglist when there are no results in the list
- if there are no results in the autocomplete taglist, the typically caret-moving key events up/down/home/end will move the caret (rather than being uselessly passed on to the empty taglist)
- all profiling now goes through a single location
- profiling now also prints extensive information on explicit code callers
- fixed db profile mode printing display for the new logger
- added new pubsub_profile_mode (AKA log-killer)
- all menu popup display and explicit memory cleanup is done through a single location
- hover windows will now not show whenever a menu is open
- hover windows will now not hide whenever a menu is open (useful when trying to go 'copy tag' off the hover taglist's menu!)
- a keyerror bug when force_idle_mode was hit right at program start is fixed
- unexpected serverside client disconnections are now caught and further request processing is cancelled
- middle clicking on an 'empty' system predicate in the a/c dropdown now will throw up the dialog to enter a value and then pass that filled in system pred onto the new search page
- duplicate parents (which can occur with overlapping grandparents) are now collapsed
- the upload pending popup message cancels properly after 'server busy' events
- whenever the client's services are iterated at the view-level, it will now be in random order, to stop sync bottlenecks choking other services (and anything else like this that might have been occuring)
- fixed a bug with double-clicking a tag selection on the the tag censorship taglist
- fixed the too-fast frame timings on some videos opencv had a problem parsing
- loading many video types will now be just a little faster
- remote files will no longer present the right-click menu option to share->copy->file
- hitting a shortcut for manage ratings from the media viewer canvas will no longer work if you have no ratings services
- the base scripts (client.pyw, server.py and test.py) now have a shebang line to facilitate non-Windows use
- laid groundwork for external client_files locations
- plenty of misc cleaning and refactoring
next week
I will continue like this. There are several old Linux-specific graphical bugs to look at and plenty of code cleaning jobs to catch up on. Beyond that, letting the client store its files across multiple hard drives is an urgent issue for many users, so I hope to add that. I know how it will work, and I've written most of the backend, I mostly just have to write some option gui stuff for it.
Version 182
win32
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v182/Hydrus.Network.182.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v182.tar.gz
I had a pretty good week. I got a lot of good work done, though not what I expected. I have decided to work more on fixes and cleanup in the coming weeks.
unicode pls
Even after last week's hotfix, there were more unicode problems. Apparently there was a (possibly long time?) problem with printing certain status updates to the log for people with unusual number localisations or service names. I now send all to-be-printed strings through a single system that will detect unicode and convert to bytestreams, so these latest problems appear to be fixed.
Please let me know if you get any new unicode en/decoding errors!
gui stuff
I went over all my custom drawing code, improving when and how certain frames of animations and ratings and so on are logistically put onto the screen. The time improvement isn't vast, but the code flow is now more sensible, with less duplicate and needless redrawing, and things that need to draw fast, like big high fps animations, are just a little bit smoother. Several previously flickery things are now all correctly double-buffered, as well.
Page Up/Down now work for the main thumbnail view.
Also, the animation scanbar will now draw a little darker if the animation is paused. Let me know what you think!
better logging
The client now uses the same new logger I recently wrote for the server, which means every object to be printed gets prepended with a timestamp, and if you happen to run the client from a terminal, the log will clone to stdout on the terminal as well.
Also, hydrus services will log every request they fuflill in the form 'PORT METHOD PATH HTTP_CODE in TIME TOOK'. This mostly means the server, of course, but the client will do it for its local file and thumbnail service and booru, if you happen to use those. I hope this helps you see better what is going on and how much CPU is being used. Also, backing up the server is easier–it now just makes a copy of the important stuff to a new server_backup folder, printing its backup progress to the log.
full list
- all printing to the log should now be unicode safe
- some other, miscellaneous file-write locations should now be unicode safe
- cleaned up some now-redundant unicode-bytestring conversion error handling
- the animation scanbar will now draw a little darker while the animation is paused
- the client now uses the same new log object as the server, so all logged data will be copied to the terminal, and all logged statements will be timestamped
- the client and server's boot and exit statements are more harmonised
- server backup is more log-verbose
- server backup now makes a simple copy to 'server_backup' folder–no more _backup nonsense
- all serverside requests will now print a line in the form 'PORT METHOD PATH HTTP_CODE in TIME TOOK' to the log
- page up and down now work (again?) for the thumbnail view. adding shift also works for selecting a page at a time
- improved some animation painting logic
- improved some rating control painting logic
- improved splash screen painting logic
- improved all other, misc painting logic–many things should flicker less or render just a little quicker
- ratings hover window will no longer re-layout (causing flicker) on ratings set
- improved some media canvas painting logic to de-jaggify some zooming
- in the manage options dialog, tag-related gui options are moved from the gui panel to the new tags panel
- added 'apply all parents to all services' option to the tags panel
- the delete key now removes active search predicates and tags in the manage tags dialog
- fixed webm link parsing for rule34hentai.net
- if you attempt to petition multiple tags, you will now be presented with a dialog asking you if you want to use the same reason for all the petitions
- edit import folder dialog now has scrollbars and will resize itself based on your monitor size
- if the static ffmpeg executable is absent from the bin folder, the client will now attempt to just call 'ffmpeg' in the normal system path
- fixed some 'None media' data calculation bugs when media viewer closed during very fast slideshow
- fixed a 'None media' mouse position bug when media viewer closed during very fast slideshow
- all wx timers will explicitly stop on exceptions (which should reduce some types of error spam)
- refactored client hydrus network session manager to a better position
- wrote a better wma/wmv determining test using ffmpeg
- refactored cv2 (OpenCV) out of the server's import tree
- fixed some server boot crash error handling
- cleaned some autocomplete matches-compiling code
- deletepath and recyclepath will no longer throw an error if the path does not exist
- server_messages folder is no longer referenced in the code–if you have it, feel free to delete it
- memory cleans up a little faster after gui page deletion
next week
I got a bit frustrated this week, as I made very little ultimate progress on my github queue and also just couldn't find the raw time to churn out an IPFS prototype. I have about 75 things in my 'this existing thing needs attention' to-do list, and that number has only been growing. I have been underestimating the time it takes to add big new things, and at the same time I still want to finish my big rewrites for the subscription system and shortcuts and gui layout harmonisation. I have felt a bit rushed for a while now, which is all self induced.
So, as I said above, I think I am going change up my schedule a bit. I will concentrate on code cleanup and test improvement and bug fixes until I am overwhelmed by new things to add rather than old things to fix. I still want to work on small new things, like simple options and keyboard shortcuts, but I will leave drag selection and mass dupe detection and adding sound and IPFS integration and streaming autocomplete results and tag clouds and ugoira support and OR searching and all the rest of the ambitious and exciting stuff until I have time and brainpower to do them well.
Please do keep contributing ideas and discussing new things–writing down good suggestions is the easy bit! But I will try to temper my enthusiasm to 'that's a great idea, I have added it to the list' more often than 'that's great, I'll start work on it right away!'.
When I am in a position to work on something big and new, I think I will open a poll with like twenty options, so people can vote on what they want.
Anyway, having said all that, I do actually want to add swf thumbs next week! It shouldn't be too difficult, as I've already found/compiled a neat exe for all the platforms called swfrender that seems to work well. I've put it in the bin folder of this release if you want to have a play with it.
Booru Logins
Hi, Hydrus newbie here.
I was wondering if there was some way I could log onto my Danbooru account in hydrus? I have a builder account so was hoping to use Hydrus to download images without a fear of a ban. I think the difference between downloading anonymously and having a member account already makes it quite a bit safer.
Would appreciate any help on the matter!
So, just making sure my dyslexia isn't acting up.
1: When set to "0", ignore mouse movements and process anyway.
2: "0" for never wait any days to vacuum?
Also, if i have my own local tags assigned to a file, will it still apply those from the repository? I have a few from booru's that aren't showing any tags other than my own.
Thanks for the great piece of software BTW!
Release Tomorrow!
I had a great week. It went generally as I hoped, mostly catching up on my oldest to-do issues, including correcting some bad webm frame timings and improving laggy autocomplete input. I also added a fairly reliable system for swf thumbnails.
With Thanksgiving coming up, I have a busy day tomorrow. The release might be a few hours later than usual.
How it's even work?
Can somebody explain in easy way how to connect to "read only file respository" using: help->i don't know what i'm doing->just set up some respositories for me, please|
dosen't seem to work. I just wan't to browse some files by tags
So I just installed this on windows 8.1 on the D: drive
The program never starts. Under task manager I get a Setup/Install icon with 1.2mb of memory and nothing else.
Ran it from the .exe What am I doing wrong?
Can we even into command line?
Hoped that this would work on a webserver
If anything I think I have free azure credit for a free GUI windows server
Release Tomorrow!
I had a pretty good week. I could not find time for IPFS, but I I fixed the latest unicode problems, fixed and added to a lot of miscellaneous gui code, and improved logging for both the client and server.
The release should be as normal tomorrow.
nonetype error
odd error im getting, the first bunch of images in my files, all turn up this error. The suigintou image selected in the picture does not, but all but a couple files to the left of and above it cause this error to start popping up, and it keeps popping up, over and over and over. Hydrus then starts to lag and act glitchy, until i click back on one of the images past the gin image, then it stops. You can see the upper image that is actually selected in the screenshot.
Here's the error message I got.
AttributeError
'NoneType' object has no attribute 'IsRendered'
File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.ClientGUICanvas", line 3359, in EventPaint
File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.ClientGUICanvas", line 3318, in _Redraw
any ideas whats up?
Version 181
win32
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v181/Hydrus.Network.181.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v181.tar.gz
EDIT: Apparently I didn’t fix the Win10 Unicode problem after all! If you still experience this bug in v181, please try this hotfix:
I had a pretty busy week IRL, so I didn't have all that much time to work on Hydrus. Nor to answer my messages, so if you have posted something to me and are waiting for a reply, please bear with me.
first, a question on windows 32/64
My Linux and OS X releases are both 64-bit, but my Windows is 32. It feels to me about the right time to switch over to 64-bit for everything, but I will either put it off or look to a dual-release if there are people still running 32-bit Windows who have no way to update.
So, if you run Win32, please let me know how you feel about a Win64 release. Just post here or send me a tweet/email. If you don't know what you are running, hit Windows Button+Pause/Break to open up the System window, and it should say something like System Type: 32/64-bit Operating System.
Win10 unicode crash
With the big unicode rewrite last week, I missed out one line–where the hydrus client asks your operating system to convert a big number like 123456 to whatever you localisation settings are, like "123,456". This worked totally fine on all my test machines. Unfortunately, some users–the reports all seem to be non-english Windows 10, but it may have affected others–have some sort of unicode character for their 'grouping comma' (or something like that–I can't replicate this myself).
Anyone affected basically had a crash/error spam anytime their gui tried to display a number bigger than 999.
I have told this OS call to always return unicode, so I expect it to work fine now. If you got this, please let me know if you have any more problems!
if you run a server on Windows, update soon!
I updated some libraries this week, and then I had to roll back a little because my test server had some very old unused and flawed sql in it that would not boot with the new version of SQLite. The code that created the bad index is long gone, so I thought it must have been some temporary thing I played with on my dev machine, but it turns out my production server had it as well, which means anyone else could have it. I don't know which version would have introduced it, but it would have been a long time ago.
In any case, if you have been running a server on Windows for a while, make sure you update fairly soon! I will be rolling out the sqlite.dll update in the Windows release on December 9th, at which point flawed server dbs that have not been updated to v181-v184 will not load!
If you happen to put it off until v185, you can retroactively update to v184 or whatever as an interim step. No damage is done by trying to connect with the new dll–it just dumps out as soon as you send any calls to the db, so the whole program fails.
drawing thumbnails better
I have iterated over most of the thumbnail drawing code, which had a few really inefficient areas. Scrolling at speed should be more reliable, and general framerate/CPU used should be better.
Also, thumbnails no longer remember their thumbnail bitmaps, which seems to have been a source of memory bloat during large imports and general browsing of 10,000+ result searches. If you now scroll through 100-odd pages of thumbnails, your client's thumbnail cache memory use will reliably cap out at 200MB or whatever you have set in the options.
taglist harmonisation
Tag lists now look more like each other and display more information. Specifically, the media viewer's 'background' taglist and the hover taglist that pops over the top of it should now show the same tag information in the same order. Neither displays count, as they always refer to one single file, but they will show (-) petitioned status.
The main 'selection tags' taglist also shows petitioned counts and collapses the counts of overlapping siblings more accurately.
And you can hit Ctrl+A on any of them as well!
full list
- fixed a potential bug in the server's db, very important you update this week if running on Windows
- improved some thumbnail page calculation logic
- improved some thumbnail page drawing logic
- fixed broken vertical resize thumbnail grid issue
- fixed some broken 'thumbnails have changed dimensions' event layout recalculation
- fixed an idiotic typo bug that was making cached thumbnail page bmps taller or shorter than needed
- thumbnail fading and other general thumbnail redraws should be a decent bit faster now
- thumbnails have handed off all bmp storage responsibility to the thumbnail cache, which should mean greatly reduced memory use when browsing very large results
- thumbnails will now only quick-draw if they have been seen before on the same page, which should reduced some page-refresh waterfall jankiness
- thumbnail media panel double buffering seems to be working better
- improved some general thumbnail drawing code
- the media viewer background taglist will now show petitioned tags
- the media viewer hover taglist should now look exactly like the background taglist (sorted lexicographically, no more counts, and now shows petitioned)
- the 'selection tags' taglist will now show petitioned tags for 'all known tags' queries
- the 'selection tags' taglist and the media viewer background and hover taglists will now count collapsed sibling tags correctly
- all taglists now support ctrl+a to select all
- improved some generic taglist value-setting code
- parents should now only expand for their specific tag services
- siblings, parents, and the pending count should now be correctly recalculated upon repository update processing
- added 'copy sources' to the import status window's right-click menu so you can copy bad urls or whatever to the clipboard
- added 'skip' to the import status window's right-click menu so you can skip urls you don't want
- added default booru for rule34hentai.net
- under certain circumstances, services with duplicate names could be created–any services like this will be renamed on update, and the loophole where this could happen is now closed
- improved a locale number formatting call to ask for unicode
- improved newgrounds parsing–mp4/wmv files should now work
- the server logger has been formalised into a proper class
- fixed some bad static image initialisation in the canvas code that was causing error spam
- fixed default import tag options for hentai foundry and pixiv
- copy->(bmp|path|local url) will now only appear as an option if the currently focussed media is local
- fixed some gallery identifier print code
- frames and dialogs will now clip their initial size to the current monitor's resolution–better scrollbar support for cramped windows will follow
- updated ffmpeg static builds
- updated sqlite command line exes
- put off sqlite windows dll update because of critical server db bug
- misc code cleanup and redaction
next week
I put off IPFS this week because of my schedule, so I would like to go for that. Also, being so behind on my emails/github issues, I'll put some time specifically aside just to write some posts and create new tasks.
When downloading from galleries based on tag searches, i have the issue of many dupes (from different uploaders). Is there a way around this? Or do I need to manually weed them out? Thanks!
Build process - Windows
Hi! In the interests of community i've decided to make this simple environment for Windows that enables debugging, building and running Hydrus from source code.
Release Tomorrow!
I had an IRL busy week, so I had less time to work on hydrus than I wanted. I did get some good stuff done though, including fixing a unicode crash bug, making a critical update for older Windows servers, writing much more efficient thumbnail memory management and drawing, and improving and harmonising the various taglists across the program.
The release should be as normal tomorrow.
Version 180
win32
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v180/Hydrus.Network.180.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v180.tar.gz
I had a great week fixing a ton of bugs, some quite serious. If you have reported something in the past few weeks, there is a decent chance it is fixed today!
middle click to open new search page
Middle clicking on a list of tags will now quickly open a page for the selected tags like right-click->open a new search page for selection. I added this to be like middle-clicking on a tag when browsing a booru. This also supports opening a page with multiple tags.
fixes to recent bugs
I have rewritten a lot of stuff in recent weeks, and while I am pleased with the new code, I did introduce some fresh bugs. I think I am getting on top of them now–thank you for reporting everything.
Last week's addition of a mouse idle check added a really bad random crash-bug for Linux (or at least anything running X11). The current theory is that I was accessing the current mouse coordinates from an improper location. I assumed it was not a problem since it was only two numbers, but it turned out differently! I have moved the code to the correct thread and I no longer get the crash on my Linux laptop. If you got this bug, I am sorry for the inconvenience. My laptop didn't seem to do it as much as other people's, so I never noticed in my normal testing. If you still get it or anything like it, please let me know!
I also recently rewrote how all predicates were passed around the client. Everything works much better behind the scenes now, except for removing system predicates from the active search. You may have noticed, if you tried to double click a 'system:duration>0' or similar in the past week, that trying to remove a system predicate with an internal value launched the 'enter the value' dialog again. This was a mistake, and is now fixed.
Also, I managed to quietly break the application of most system predicates in searches where tag or namespace predicates also exist. If you had a query of 'series:star trek' and 'system:mime=flash' or whatever (as one reporting user had), the mime predicate was not being applied. This too is fixed.
unicode and paths
Unicode (bits of text with unusual characters) has proved a continuing pain, and something I didn't wholly understand in python, so I decided to read up on it properly this week and go over almost every single line that deals with it. I am really pleased with the result, as several awful old bugs based on my misunderstanding are now fixed. If you have recently seen any UnicodeDecode errors, they should now be gone (or replaced with different, hopefully more helpful errors!).
I've also gone over all my interactions between hydrus and the OS's file system and made sure that everything is correctly converted to and continuously handled as unicode, which should mean that no instance of importing or exporting files with japanese characters or anything else unusual should be a problem any more. Also the basic way that bits of file paths are constructed has been moved to a more proper method, so if you have had any errors about odd, non-existent paths, please let me know if that is fixed now.
full list
- middle click on any taglist now does 'open a new search page for selection'
- moved mouse idle checking code to the wx thread, which seems to have fixed a serious X11 crash
- memory gunk should now be painted over when transparent animations are resized on linux
- fixed some page up/down index calculation on the autocomplete dropdown
- system predicates will remove themselves from the active predicate list silently again
- system predicates will work again in queries that include tags!
- greatly improved my string/unicode conversion code
- dropped many inefficient unicode conversions from db code
- improved the sql-grammar of some specifically efficient db code
- removed incorrect unicode conversions from url/network request code
- removed unicode conversion code from network response code
- all directory iteration should now occur in unicode
- all inputted paths should now be converted to unicode
- replaced every single ugly raw path concatenation with a better method
- improved some bitmap placeholder initialisation that was causing Linux terminal spam
- added an option to always hide the preview window to options->gui
- the preview window will not accept setmedia calls if it is hidden
- if the preview window is actively hidden due to an unsplitting event, it will null its current media
- fixed a shutdown calculation that was under-remembering the height of the preview window by four pixels every time
- improved thumbnail waterfall code–it should be a little faster. less subject to lag when taxed, and waterfall with more genuine randomness
- improved some thumbnail page drawing and clean/dirty conversion code
- when pages are dirtied, any to-be-waterfalled thumbnails are cancelled, making superfast scrolling less crashtastic
- new dirty thumbnail pages are now generated as needed, rather than precached
- subscriptions will now correctly do a polite wait after fetching tags for redundant files
- splash screen will now explicitly raise itself to the front on init
- the export tag txts function of the export dialog now exports the tag txt to [filename].jpg.txt, rather than [filename].txt
- the server now stores its updates in service specific subdirectories, like the client
- some importers will better report when they are waiting politely
- OS X (and anything else with OpenCV 3.0.0) now uses OpenCV to render gifs by default, falling back to PIL on error (like only Windows used to)
- 'open externally' from thumbnails now defocusses the preview window
- refactored away a bunch of bad global controller references
- added some shutdownexceptions
- silenced shutdownexception terminal spam
- reworded the 'maintenace and processing' options panel a little to explain things better
- improved how the base_dir is figured out on program boot
- improved serverside 'should I clean dirty updates?' check code
- deleted some old code
- split path-specific functions off from HydrusFileHandling to HydrusPaths and de-looped my import tree appropriately
- general refactoring
next week
I feel better than I did two weeks ago about the bugs in the program, so I think I might have a go at adding some sort of IPFS support. At the least, I will make a proper plan. Other than that, I would like to make sure the media viewer's contradictory background/hover taglists actually say the same thing, and that that thing is useful, and I'll look at making things fit better on small screens.
Suggestion
I recently discovered this great tool. It is similar to a system I have been working on. I would like to contribute an idea from mine for you to consider.
Instead of storing the files on disk as it does it can divide data into fixed sized chunks and encrypt those. Doing this would prevent other processes on the system from scanning the files to recognize file hashes or deduce information from file sizes. This way all the data can be stored on a shared disk drive or cloud drive and only the database needs to be stored in an encrypted container.
So, I may be an idiot and can't figure this out or this may be a decant suggestion.
I like to just browse my pictures, but I only have two options if I want to do this. 1. type in a tag and load the tag or 2. load my whole database.
Option one loads fast but I'm obviously limited to one tag, option two works great but I end up waiting a good 5 minutes before the db loads in fully.
So, I was wondering if there is a way to just have a master list of my local tags so i can browse by subject without loading my whole database.
Release Tomorrow!
I had a great week. I mostly fixed bugs, including a persistent crash for many Linux users, adding some better animated gif support, and removing the dialog spam when trying to remove system predicates from the active search.
The release should be as normal tomorrow.
Version 179
win32
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v179/Hydrus.Network.179.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v179.tar.gz
I had a good week. I have done a bunch of small things and added multiple tag selection and interaction to all the tag lists.
multiple tag selection
Anywhere you see a list of tags or system predicates should now support multiple selection. You can use ctrl- or shift- to select what you want, and both the mouse and the keyboard should do it (e.g. Shift+End will select everything from the current position to the end of the list).
Furthermore, any tag window that lets you 'activate' a selection, like hitting enter or double-clicking to choose an autocomplete tag result, should now activate everything you have selected in one go! So, if you have some thumbnails that have several character tags in the 'selection tags' box on the left, you can select two of them with a quick ctrl+click and then double click to throw them both up to the 'search' predicate box at the top in one step. Have a play with this to get a feel for it–I was already doing it during unrelated testing today, and I really liked the feel of it.
In updating the various methods that pass tags to and accept tags from these lists, I rewrote several related things–double-clicking the tag list that hovers over the media viewer will now launch the manage tags dialog, for instance, and when you see parents beneath some autocomplete tag results, they will be selected in a block along with their child. The tag right-click menu also supports your multiple selections, whether for copying to clipboard or initialising the manage siblings or parents dialogs with several tags at once.
I am pleased with this, but due to all the changes, I expect there are some bugs. Please report any tracebacks or odd censorship/autocomplete/sibling/parent/whatever behaviour you get. There is also some more stuff I didn't have time for this week–I would like to single->multiple-ify the internal workings of the manage tags dialog so you'll be able to assign one petition reason for multiple tags in one step, and I will add Ctrl+A and Delete key support to all the tag lists where appropriate.
mouse idle
Cancelling idle mode (when the client does high-CPU stuff after a period of inactivity) can sometimes be difficult, because the client lets itself become so busy that it doesn't have time to notice that you want to do something with it! If you have ever been unable to un-minimise the client after a while, this is usually what is happening.
So, I have written a simple 'when was the last time the mouse was moved?' test to complement the idle mode calculation, so it basically acts like a screensaver. By default, it is set to disallow idle mode if the mouse has been moved in the last ten minutes, so if you are using your computer for anything at all, your client's heavy cpu will probably turn off. You can edit this timing, including turning the check completely off, in file->options->maintenance and processing.
hentai foundry 'count' error fixed
If you paused your hentai foundry subscriptions because the gallery parse step was throwing up an error about 'count' something, you can turn them back on again! Some new 'a' html tag without an 'href' attribute was popping up for some reason, and my parser wasn't expecting it, but now I've fixed it.
full list
- all tag listboxes support multiple selection
- all tag listboxes manipulate their data as sets of tags rather than individual tags
- all tag listboxes report to their callbacks as sets of tags
- many dialogs and other windows that use tag listboxes now deal with tags as sets
- double clicking the media viewer's tag hover window now launches the manage tags dialog
- the media viewer's tag hover window now includes (+1) counts like the manage tags dialog–harmonising the underlying canvas tag list will follow
- lots of tag/sibling/parent code has been iterated over
- ditched some misc redundant code and needlessly tightly coupled object relationships
- some bizarro tag sibling dialog code has been rewritten
- the autocomplete dropdown for 'writing' will now not expand search results to include parents when the receiving control isn't interested in them or it is otherwise not appropriate
- when the autocomplete dropdown does have parents, they will be selected in a group with their child
- the autocomplete dropdown for writing now broadcasts 'I'm done with tagging, close the dialog pls' in a better way
- 'activating' the sibling or parent dialogs' listcontrols (usually by hitting enter) now processes all rows that are selected, not just the first
- manage siblings and parents dialogs can now take multiple initialising tags from the taglist right-click menu
- removed some redundant listbox code
- cleaned up and improved some tag list event processing
- cleaned up some taglist right-click menu code
- added system-wide mouse idle test to idle calculation. you can set this in files->options->maintenance and processing, and it defaults to ten minutes
- import folders now support tag import options' explicit tags for multiple tags to any tag service
- existing import folders will be updated to the new version, and if a local tag exists, it will be intserted into the new import tag options
- fixed a hentai foundry page parsing bug
- the deviant art downloader can now download >1024 pixel width versions of images via the download button (and a bit of cookie magic)
- gallery page queries that 404 (like for a non-existent username) will now report 'Gallery 404' rather than spamming the gallery's whole custom 404 html page to the status box
- fixed a bad layout flag that meant some namespace checkboxes in tag import options could remain hidden during first panel expand until first mouseover
- the import status frame now initialises its status text properly
- fixed last week's num_tags optimisations, which accidentally broke num_tags < x for x > 1
- the client should work on both OpenCV 2.4.x and 3.0.0 (thankfully, the only difference for our purposes turned out to be some static variable renaming)
- the Windows and OS X releases now come with OpenCV 3.0.0
- increased the max period of import and export folders to 30 days
- the launchfile/directory thread is now a daemon, so the (some flavours of Linux) client can shutdown even if an externally launched file/dir remains open
- misc cleanup
next week
This week, I actually spent more time working on the stuff I didn't highlight, which was mostly boring-but-semi-tricky bugfixes. I think I would like another week doing that quality-of-life stuff before I start anything big and new.
I've been leaching off gelbooru for a while
What can be done so I have access to it's tagging system?
Seeing as I'm only scraping it's character/series/etc. I'm not up for doing it manually
Version 178
win32
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v178/Hydrus.Network.178.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v178.tar.gz
I had a good week mostly fixing bugs and catching up on small cleaning jobs.
explicit tags for tag import options
You can now tell any instance of tag import options (the expandable/collapsible panel where you select which namespaces to parse from a gallery site or whatever) to add specific tags to every file imported. I am calling these 'explicit tags', and you can edit them with a little button that is now just below the list of namespaces. This is useful for a subscription where you might want to add 'check these later' as a local tag, or perhaps 'creator:blah' for a creator's tumblr (where there is otherwise no creator tag to parse).
You can also set up default explicit tags in file->options->default tag import options.
I will swap out the 'local tag' option in import folders for an unnamespaced tag import options panel next week!
several bug fixes
Gelbooru 503 spam is fixed!
Pixiv tag parsing is fixed!
Drag and drop for thumbnails is much more responsive!
Weird gifs will animate better!
The manage ratings dialog will stop spamming erroneous 0/10 ratings!
Video import is fixed!
OS X won't crash when you use the manage tag dialog's previous/next media buttons!
full list
- import tag options now supports 'explicit tags', which will be added to all files imported
- the default import tag options system supports explicit tags, and will propagate them to new import tag options objects
- the gelbooru 503 Forbidden problem is fixed
- all gallery downloaders now send the page_url as http Referer header when fetching their image_urls
- pixiv parsing now grabs the correct tags for the image! previously, it was fetching the user's most common tags
- for now pixiv manga (which is not yet supported), will be skipped over in the gallery parsing step
- fixed a thumbnail drag-and-drop calculation that meant the DnD event would not start if the initial mouse movement went in an up-left direction
- bad animated gif frame timing detection is improved
- the fallback animated gif frame timing is scaled back to a more reasonable 12fps
- fixed a critical bug in the manage ratings dialog that meant any NULL or MIXED numerical rating controls were mistakenly creating 'set rating to 0.0' content updates
- fixed an obselete call with the new Pillow library that broke video thumbnail generation
- added an explicit file import test to my test suite (previously, this was a manual test, and I did not notice the failed video files last week)
- fixed a critical os x crash after navigating media with the manage tags dialog launched from the media viewer
- improved speed and memory use of num_tags queries a little bit
- reordered the predicate filters in the main query code so that num_tags happens right at the end, where it will run fastest
- removed an unnecessary filter from the main query code
- removed an accidental doubling of the simple predicate filters from the main query code where the file service domain is specified
- improved an important db index and made sure the db uses it everywhere it can. it should speed up most non-'all known tags' queries
- harmonised how all the new serialisable JSON objects are sent to and read from the db to a handful of simple general functions
- externally opening a file is offloaded to a temporary thread to avoid gui lockups on OS flavours that have delayed 'open this file pls' calls
- increased the commonly used read/write 64k block size to 256k, let's see if it reduces some file fragging
- clientside hydrus bandwidth reporting is a bit faster and simpler
- general code cleanup
- general refactoring
- a long-term refactoring job that aimed to better decouple the client, server, and hydrus parts is now completely done
- delineated the layout of the gallery and page of images management panels a little
- updated several links in the help and the program to https
- the petition content checkbox list is now significantly taller
- cleaned up the static folder a little bit
- silenced an unhelpful webm last frame error
next week
One of the video rendering libraries I use, OpenCV, has a major new version, which I will attempt to upgrade to for all platforms. After that I think I would like to look at adding multiple selection to my various tag lists, so you can remove/petition multiple tags in one step, for instance, or conversely add several tags from the autocomplete dropdown. Otherwise, I have lots more small things to do, a bit like this week again.
Release Tomorrow!
I had a good week. I improved, fixed, and updated a lot of small things, and I completely rewrote the data-handling behind the lists of tags throughout the program, so they should all now support multiple selection (and subsequent action, so you can now select several tags and create a new search page with all those tags, for instance).
This taglist stuff spiralled into a much larger job than I expected, so I still have a little work and a lot of testing still to do. I am confident I can get things generally working tomorrow, but the release might be a couple of hours later than usual.
Minor improvement suggestions
1) Please make capitalization consistent while the option is enabled. Better yet, do it by default ;)
2) The viewer should not say "hydrus client fullscreen media viewer" since using it in full-screen mode is not required.
Large Imports
So I'm getting started with Hydrus. So far, I'm not ready to completely transition to it when it comes to my image management, so I don't really mind having two redundant collections since I've got HDD space to spare. As it currently stands, I've got around ~50k images I wish to import. Now, it's still at it and I've seen memory usage steadily increasing to the point it might end up crashing and I'll just have to start over.
My question is, is there something I'm missing when it comes to such large imports? It'd seem like even though a given image is already in the db, its data is kept around in memory. I could try to split things up somewhat but it'd certainly take me a while, I just want to know whether or not it'd be viable (even if extremely unwieldy) to import my entire collection whenever I just want to "sync" with my Hydrus database and expect it to skip over any files it already has included without much memory thrashing.
Hydrus docs and HTTPS
Could you please update your docs with HTTPS URLs such as the MediaFire and 8chan links (others too)? Thank you.
Version 177
win32
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v177/Hydrus.Network.177.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v177.tar.gz
EDIT: I accidentally broke video thumbnail generation, so video imports don't work in this! It will be fixed next week!
I had a good week finishing up a little of last week's work and improving some admin stuff.
default tag import options
All new instances of tag import options should now be prefilled as you have set in file->options->default tag import options. This was occasionally called advanced tag options before, which was the old way of storing that data, but now everything should be switched over to the new system. Any previous defaults you have set should be converted over as well, but make sure to check for yourself.
faster petition processing
(This is only for people who run their own repositories.)
Petitions are now decided in batches that share the same petitioner, action, and reason. For instance, if someone submits twenty sibling petitions all with the same reason, you'll see them all at once. There's a list of checkboxes for every row of content and a single 'process' button to act on the whole list in one go. Overall workflow lag is also reduced. This should make going through your queues a bit easier!
This involves a change to the network protocol. I would usually increment the network version and force everyone to update, but since this only affects admin stuff, I will leave it up to you. A v177 client cannot process a v176 server's petitions and vice versa, so make sure you update your servers this week.
full list
- the old YAML "default advanced tag options" is converted entirely to the new JSON "default import tag options"
- created a new JSON object to store new options objects
- updated the gui to handle new default import tag options objects
- existing default advanced tag options data will be converted to the new system on update
- the petition management panel now presents similar petitions from the same user together, with the relevant data rows in a checkbox list, so janitors can approve/deny many similar petitions all at once
- the petetion management page no longer auto-loads file results for a petition–you can now see specific results for each sub-petition by double-clicking on them
- the server-to-client petition object is converted from yaml to json
- the new petition object now stores all the different 'content' rows for the same service/account/reason/action
- small content rows are now stored and transmitted in their own object
- the network now supports POST JSON requests
- account modifications now work through JSON
- the account identifier object is converted from yaml to json
- in the path tagging dialog, regexes that do not compile will now say so, reporting their short regex error statement
- fixed a bug that could sometimes occur when cancelling an ongoing gallery query
- cancelling an ongoing gallery query is now reflected in the gui much faster
- repositories will not burn CPU cleaning their updates if there are still pending file/mapping petitions to process
- repositories will clean updates in smaller bursts to reduce the contiguous server_busy time
- fixed drag and drop thumbnail ordering, which actually wasn't working but was accidentally randomly sorted when I tested it!
- a certain kind of error during vacuum that can mean two different things is now caught, and automatic vacuums then suspended while the user figures out what is going on
- revised some v176 update code that wasn't parsing some downloader pages correctly–now, if a downloader page cannot be updated, it will be discarded
- updated to Pillow 3.0.0 on all platforms and fixed some obselete image calls
- general code cleanup
- general code refactoring
next week
I would like to extend tag import options to support "add tag x to service x for all imported files," and there are several bits of code cleanup I want to get caught up on, just so they don't linger. There's also some broken gifs someone sent me that I want to get a proper look at.
Release Tomorrow!
I had a good week. I cleaned a lot of code, optimised some important db queries, and fixed quite a few bugs.
The release should be as normal tomorrow.
Someone should really make a video tutorial on this. For us learning disabled folks. Awesome project BTW, I'm enjoying it thus far.
Hydrus crashes on exit from fullscreen
I am running Hydrus on OSX v10.10.5, and every time I am tagging in fullscreen and I leave it, Hydrus will crash. This does not occur if I have not done any tagging, and have only gone into fullscreen and left it, nor does it occur when I have been tagging outside of fullscreen.
Release Tomorrow!
I had a good week. I fixed default tag options for the new system, greatly improved janitor-side petition processing, and fixed several bugs.
The release should be as normal tomorrow.
Should make it so you can select multiple tags from tag picker / image viewer to add siblings/parents to multiple entries at once
Can't add parents back
I removed some tag parents and realized I made a mistake, but when I went to add one back I get this when I push the add button in the parents dialog:
NotFoundException
Data not found!
File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.ClientGUIDialogsManage", line 7671, in EventAddButton
File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.ClientGUIDialogsManage", line 7576, in _AddPairs
File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.ClientGUICommon", line 4785, in GetIndexFromClientData
Hydrus newbie
Hi, I'm new to Hydrus.
In the past I used a downloader to grab around 100GB worth of files from gelbooru. I was wondering how easy would it be to import them into Hydrus and have tags automatically assigned to the images.
According to http://hydrusnetwork.github.io/hydrus/help/advanced.html under "tag archives", I just download the gelbooru.7z, drop it in "\db\client_archives", and let the magic happen?
Im just curious how well this will work before I spend +10 hours importing files.
Another quick question, how do I force start a subscription? I accidentally paused it and would like it to continue. Do I have to wait a day for it to start? It would be neat if there was a way to review the subscription process, so I can get an idea of how long it'll take
Version 176
win32
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v176/Hydrus.Network.176.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v176.tar.gz
I had an excellent couple of weeks rewriting the gallery downloader page and the entire subscription system. I am glad I took the two weeks, as I only finished it yesterday afternoon. Your existing gallery pages and subs will be converted on update.
galleries
Gallery pages and subscriptions now use the new downloader engine. Gallery pages can be saved to gui sessions and should remember their file and tag options and also current search statuses, even down to exactly which gallery page they are currently on. Both also now have the new 'import status' button, so they recover much better from failed files and then let you retry them. In general, they load and save and work a bit faster. More information is displayed in a more efficient space.
Furthermore, gallery pages now support 'get tags even if file already in db' , and subscriptions are a bit easier to configure and have a new optional file limit on the number of files downloaded every time they run.
I am very pleased with the result here, but I have changed a lot of code. While I am fairly confident there is nothing very broken, there are likely a few bugs floating around. Please let me know if you notice anything not working how you expect!
In switching everything over, the Hentai Foundry filter options (where you can select the 'show yuri' filter and all that) proved a (continuing) pain in the neck to keep track of. I thought about writing a seperate class just for HF so I could remember this info and the respective tracking cookie, but I decided to keep the new code simple and non-hacky for now and made HF searches completely unfiltered and always sorted 'newest first', so they now always work like every other gallery hydrus supports. If people want content filtering, I plan to add tag-based filtering in future that should plug straight into the downloaders (so you can say "never download anything that has the tag 'blah'").
I may bring back hentai foundry 'popular/views first' sort for tag search if people want it, but each new sort will have to count as a separate gallery to select.
Also, I did not have time to convert 'default advanced tag options' over to the new system, so when you create a new gallery page or subscription, your import tag options will not be pre-checked.
full list
- the gallery downloader is updated to the new system
- the gallery downloader code is generally a bit better all around
- the gallery downloader will now remember its urls and file and tag options and paused status and so on throughout a reboot
- the gallery downloader supports 'get tags even if file already in db' option
- the gallery downloader management panel is updated for the new object. like the other panel updates, it is a little more minimalist
- the gallery downloader management panel now has the import status button so you can review your parsed urls and retry them, or copy errors, or whatever
- existing gallery downloader pages in gui sessions will semi-convert on the db update
- subscriptions are updated to the new system
- the subscription code is generally a bit better all around
- subscriptions will no longer dump out on a failed file!
- subscriptions can now recognise when files are uploaded during a page-walk (which shuffles everything up by one, of course), and recover without assuming it has synced
- subscriptions can now be set an initial file limit and a periodic file limit separately
- the manage subscriptions dialog is updated for the new object
- the manage subscriptions dialog now has the import status button so you can review your parsed urls and retry them, or copy errors, or whatever
- existing subscriptions will convert on the db update
- subscription popup now shows per-file download progress
- boorus that use number of thumbs to advance their gallery page indices can now resume
- the thread watcher now obeys 'exclude deleted' status more reliably
- the 'page of images' downloader now obeys 'exclude deleted' status more reliably
- subscriptions now attempt to recover their sync status when a new file is uploaded during sync
- fixed a bug where the thread watcher was not applying the filename namespace to new files, only redundant ones
- the old advanced_import_options dictionaries are all updated to the new import_file_options object
- improved subscription-dialog-was-opened-while-subs-were-running detection and reaction
- the way galleries are identified behind the scenes is now much simpler and more flexible
- for now, hentai foundry downloads and subs will no longer support the specific hf filter–they will be unfiltered and sorted newest first, so they work like all the other galleries. I may reintroduce 'sort by popularity' if there is demand for it
- a bit of importer status update refactoring and cleanup
- fixed some db vacuum maintenance timing that was interfering with gui reporting, so the 'currently vacuuming' popup never appeared until the vacuum was complete
- fixed a possible bug for import folders that have a local tag set
- splash screen is no longer stay_on_top–if there is demand, I will add an option
- fixed a bug in the page deleted detection code
- added page hidden detection code, replaced some ambiguous pause code with it
- improved pause/play code throughout the program
- corrected some bad x/y status texts
- for simplicity, the import_file db command now always returns ( status, hash )
- for simplicity, the import_file db command no longer takes service_keys_to_tags
- plenty of general code cleanup
next week
I will get 'default advanced/import tag options' working, and I already have plans to extend this new stuff to support a button for 'sync this sub on dialog ok', the ability to choose individual gallery streams (e.g. do you want just a Hentai Foundry artist's pictures or scraps, rather than both?), and adding a control for 'add tag x to service y for all files imported' to all instances of import tag options, which several people have asked for specifically for subscriptions.
Beyond that, I think all the import systems are converted to my new system, so that big job is completely done and I am free to think about a new big thing. First of all, I really want to rewrite the petition fetch/approve/deny system entirely, as I have let about 500 petitions pile up on my public tag repo because the current system has a terrible workflow. After that, I may look at adding audio support or moving towards mass 'looks very similar' dupe search, or adding text/html/shift jis file support, or just doing a whole load of small things. Let me know if you have something particular you would like!
I am behind on my messages and emails, so please bear with me for a couple of days while I catch up.
Possible Version 176 oversight
just a heads up, default tag import options don't seem to work on the gallery downloader anymore.
So you have to manually check the boxes of the namespaces you want to import along with the files downloaded each time you open a gallery download tab.
Version 175
win32
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v175/Hydrus.Network.175.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v175.tar.gz
I had a pretty good week, but I was fairly short on time. This release has several small improvements and fixes. I expect the next release to be in two weeks.
regex favourites
The 'regex shortcuts' button in the path tagging dialog has a slightly reformatted menu that now allows for saving your favourite regex phrases. They will copy to clipboard like the other entries, and you can also set a description for them.
custom animation start point
I had previously heard a joke that all youtube video could be safely skipped by ~20-30%, but apparently this has a semi-formal definition called the "Wadsworth Constant". So, under options->media, you can now specify where animations start by default. You can set whatever percentage you want.
When I move to animated thumbnails for gifs/video, I'll have them obey something similar to this, which should alleviate the common problem of having a page full of black thumbnails for video.
full list
- the 'regex shortcuts' button in the path tagging dialog now permits the managing of favourite regex phrases
- added 'start animations this % in' options to options->media
- animations will no longer clear to white while scanning beyond their immediate frame cache
- modifier-click (like shift-click) will now drag animations without altering their pause/play status
- the splash screen is a little larger, to better fit db update text in
- the splash screen's text is split into two lines, and shutdown repository processing is a bit more verbose
- fixed some incorrect dialog text in manage sibling and parent dialogs
- rewrote sibling and parent pair update code to deal with unusual situations more gracefully
- improved how the new downloader engine saves error notes
- the import review frame now lets you copy the selected entries' notes to the clipboard
- video files will no longer fail to import in windows because of unicode characters
- recycling files with unicode characters on Linux will work more reliably, and when it breaks, it will fail more gracefully
- the client and server can now detect and recover from a rare error where a database file is created but never initialised with the appropriate tables
- the tag/file upload popup message now dismisses itself after a few seconds, rather than pointlessly lingering
- wx event ids used in one-shot menus will be recycled, which should alleviate the 'after a long time, some menu selections stop doing anything' problem
- some code cleanup
next week
I am now going to concentrate on converting gallery downloads and subscriptions to the new downloader engine, where they will generally work better, allow for download reattempts, be easier for me to customise and extend, and survive through client restarts. This has been a long time coming, and it doesn't look like it will be simple or quick. I expect it will take me two weeks, although it may take more or less. I will make a post next Tuesday with an update, and then hopefully a 'Release Tomorrow!' post the week after.
Please keep sending your bug reports and thoughts in as normal, but unless something is an emergency, please expect a short delay before I can work on anything else.
Release Tomorrow!
I had an excellent couple of weeks. I rewrote my gallery downloader and subscriptions systems to use the new downloader engine. Both now work more reliably, support a few extra features, and will be much easier to extend in future. Existing downloader pages and subs will be converted to the new system.
I still have a bit of testing to do, and a busy IRL day tomorrow. The release should be as normal, but perhaps a couple of hours later than usual.
No release tomorrow!
As expected, I am now hip deep in the gallery downloader and subscription rewrite. It is going a lot easier than I thought, and I am very optimistic about the finished product, but there remains a surprising amount to do.
I will keep hammering away with the aim of getting a release out on Wednesday the 7th. I will post another update on the 6th!
Hmm
Is it possible to make it so hydrus client/server is the only way to access the images on the database?
Might not be limited to make the software programmed to do so, but other methods that use windows functions.
Would allow of hiding images a little better from computer crawlers (software/malware/cops)
Technically Advanced People Go!
Release Tomorrow!
I had a good week, but I was short on time. I added regex favourites and made several small additions and bug fixes.
The release should be as normal tomorrow.
Can't import files with Japanese/Chinese filenames
It just fails immediately, I'm guessing the function to open the files can't handle the character set. Python didn't get proper Unicode support until version 3.
If I rename the characters out of the files they import fine. Anything you can do about this?
High-DPI font fugliness
I have my fonts set to 125% DPI mode in Windows and many words are squished off the sides of dialogs in Hydrus. Can you look at improving this?
Version 174
win32
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v174/Hydrus.Network.174.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v174.tar.gz
I had a good week. The server starts and shuts down in a neat way, and I did a lot of small things for the client, mostly gui stuff. None of that stuff particularly needs explaining, so read the full list if you are interested.
server
The server now supports stop and restart. As part of this, its admin service accepts empty POST '/shutdown' requests from localhost. If you can make use of this from another application, have fun with it! I plan to add similar http POST requests to the client for importing files and adding tags that you'll easily be able to plug a scraping script or whatever into.
The server will now also copy log output to the console, so you can see errors and other status updates as they occur. I've generally neatened these messages as well, including adding timestamps to everything. I will probably add simple request logging as well, in future, and so you can better see what is going on and diagnose any problems.
full list
- 'server stop' and 'server restart' now work!
- server will now log everything to console as well as server.log
- server log entries will always be prefixed by the time they were made
- server shutdown will close services more neatly
- server shutdown cannot be triggered multiple times with multiple keyboard interrupts
- server initialisation and shutdown is more verbose
- server now accepts POST '/shutdown' requests from localhost
- fixed a server shutdown bug
- mousewheel events that hit the command and ratings hover windows will be propagated to the media viewer (e.g. if you click a rating, you can now scroll immediately to move to the next file without having to refocus the media viewer window)
- fixed a bug where zooming out would go immediately to 1% if the media's exact fit zoom was already in the list of default zooms
- added 'check db integrity' to database maintenance menu
- multiple drag-and-dropped thumbnails will now be listed inside the drag-and-drop object in their hydrus thumbnail order (was previously random)
- animations will now pause/play correctly during inbox filtering
- most client errors now cause a 1 second wait on behalf of the caller, which should bottleneck error spam when it happens
- fixed some buggy display and rescind behaviour with pre-dialog-existing petitions in the manage parents and siblings dialogs
- the manage parents and siblings dialogs will now not attempt to remove/petition/rescind a pair if the entered list of pairs has entirely new entries
- the manage parents and siblings dialogs will now attempt to maintain their sort order after you change some pairs
- generally cleaned some parent and sibling dialog code
- added option to options->files and trash for whether to use your OS's recycle bin
- system:mime text now specifies specific mime group if appropriate, or otherwise lists every mime if >1 selected ('system:mime is specified' is dropped)
- hydrus repositories will sometimes catch and recover from zero-length update files
- hydrus repositories will catch processing parsing errors and pause when they happen
- the new hdd, page of images, and thread watcher import daemons will tidy themselves up if their accompanying pages are completely deleted
- improved how daemons shut down
- improved client shutdown logic and speed
- some misc code cleaning
- more general code cleanup
next week
I have some IRL stuff to deal with, so I might be short on time. Otherwise, I mostly want to properly start my gallery/subs rewrite and add regex favourites to the file-path-tagging dialog.
Release Tomorrow!
I had a good week. I added stop/restart to the server, improved some gui behaviour and error handling, and fixed some bugs.
The release should be as normal tomorrow.
Wadsworth Constant video preview
Can we have an option to start video playback 30% in or a quick jump key?
Version 173
win32
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v173/Hydrus.Network.173.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v173.tar.gz
I had a very good week. I converted import folders to the new system and I added/improved some gui stuff.
import folders
Import folders now work faster, do more, and break less often. You can pause them individually, filter imports by mimetype, set multiple imports for the same location, have them run completely silently, and set separate actions (delete, move, or ignore) for files that were successful, redundant, previously deleted, and failed.
Check out the new edit dialog under file->import folders for everything new you can do. Your existing import folders will be pseudo-converted to the new system and set to a paused state, so check they are set exactly how you want after you update and then unpause when you are happy.
mime selection
I've written a better 'mime selection' control for import folders, so I've copied it over to system:mime. You can now search for any combination of mimes, and set the default in options->default system preds as well.
misc gui stuff
Animations will now pause/play if you click on them! The animation scanbar beneath no longer does pause/play–the animation will scan about as you drag, and when you let go, it will only play again if it was playing before you started scanning. This is a huge improvement for me, but let me know if you find it bad and I'll add some options. EDIT: I just discovered that this new pause/play clicking triggers inbox filtering actions, which was not intended–I will fix it next week.
There are some new options. You can roughly throttle downloads with the 'polite wait' option in options->downloading, and you can disable the confirm dialogs for trashing and archiving files under options->gui.
The manage tags dialog now has a 'remove all tags' button that should make fixing automated mistakes a bit easier! Let me know if there is some combination of tag states that it doesn't work for.
full list
- converted import folders to the new system
- import folders now have names, so you can have several pointing to the same location
- import folders can filter by mime
- import folders now have separate actions for successful, redundant, deleted, and failed files
- these actions can be 'leave alone', 'delete' or 'move to location'
- import folders now support import file options
- import folders can now be individually paused
- import folders' popups are individually optional
- import folder edit dialog provides a path cache button, like new download pages, to let you review the paths it is remembering
- existing import folders will be roughly converted to the new system on update, but they will be in a paused state, so you can check they are what you want
- created new mime selection control
- mime selection control added to mime system predicate selection dialog, so you can now search by mime more specifically
- updated some mime system pred code to deal with this change
- a single non-dragging left-click on an animation now pause/plays!
- the animation scanbar will now only resume playing the video/flash after a scanning click/drag if it was playing before the drag began.
- added a 'polite wait' option to options->downloading. this is temporary until I write a better clientside bandwidth throttling system
- added a 'remove/petition all tags' button to the manage tags dialog to make mistakes easier to clean up
- added 'confirm send to trash' to options->gui
- added 'confirm archive/inbox' to options->gui
- added a simple 'export tags in .txts?' button to the export dialog. In future, I will expand this to do service-specific exports and also add it to regular export folders
- fixed some invalid name error handling in the manage dialogs
- fixed a bug that sometimes occured at the end of an admin immediate sync
- fixed a rare shutdown-order error in video rendering
- fixed an animation bar initialisation error that was causing error spam on some videos/flash
- general import code cleanup
- cleared up admin-side petition wording and presentation
- improved the threaded workflow and error reporting of the admin-side petition page
- the thread watcher now reports its number of next seconds until the next check in properly formatted time
next week
I couldn't find time to do server stop/restart this week, so that's first. After that, I really want to get started on the subs/gallery downloader rewrite to the new system. This will be a multi-week job, so I want to see if I can do it in steps. Other than that, I have plenty more small things to do.
Release Tomorrow!
I had a really good week. I mostly rewrote import folders to the new system and significantly expanded how they work. I also added quite a few quality-of-life gui things, from a new control for selecting mimes to options for the trash and archive confirmation dialogs to making animations pause/play on a normal click.
The release should be as normal tomorrow.
instead of copying the file into db folder, is it possible to just refer a external file in db?
Version 171
win32
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v171/Hydrus.Network.171.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v171.tar.gz
I had a great week, spending most of my time making the laggy parts of the program less annoying.
isalreadyrunning on linux
The 'new and improved' program-is-already-running detection code was misfiring for the Linux executable due to some odd pid behaviour. I have fixed it.
maintenance and processing
From now on, the client will not do its CPU-heavy stuff while you are doing searches and browsing media. This includes general database maintenance and repository processing. The client will wait until it is idle, or it will try to run them for a bit on shutdown.
If you happen to catch the client while it is doing some tag processing, it should detect you are back faster and automatically stop working on the processing in a few seconds. Ideally, you won't have to think about processing any more–it will just happen in the background. This is especially true if you leave the client on all the time.
Furthermore, I have moved repository processing to a lower, faster db level. I get anything from 1000-8000 rows/s on my dev machine. I am not sure how fast a real client with 100,000+ files will go, but it should be at the very least twice as fast as previously.
I have shuffled some file->options things around a little because of this. You can customise the idle and shutdown behaviour in the new maintenance and processing option page.
Having sync only run in the background is a big change, and I tried this this week as a bit of a test. I am really pleased with the speed increase it allows, but I think I may want to revisit the exact workflow. I might add a button on review services for 'sync now' or something. Maybe my default idle options should be something else. And maybe I need to write some more help, for new users. Anyway, let me know how it works for you!
full list
- improved isalreadyrunning code
- fixed isalreadyrunning code for the Linux frozen build
- the client's isalreadyrunning splash check will respond more quickly if you choose to wait
- repository processing sync will no longer update the frontend, which approx doubles its speed
- repository processing sync now only occurs when the client is idle, so it won't slow your browsing down
- if you catch the client while it is processing, doing pretty much any action will make it quit out nice and quick
- repository processing sync now occurs at a lower db level, meaning less laggy popup ui update and overall faster processing time
- repository popup disappears after a few seconds once it is done–if you leave your client on all the time, you probably won't see it again.
- split 'maintenance and memory' options panel into 'maintenance and processing' and 'speed and memory'
- added 'cpu busy' option to options->maintenance and processing
- added 'run stuff on shutdown' option to options->maintenance and processing to set whether pending db maintenance and repo processing can happen on shutdown
- added 'max minutes on shutdown' option to options->maintenance and processing to limit how much shutdown processing can be done in one sitting
- repositories can now sync on shutdown–they will report through the splash window
- shutdown maintenance is checked and run in a better way
- moved to a two-stage shutdown procedure
- daemons are better kept track of, and they talk to the controller in a better way
- daemons shut down in a better way
- improved a bunch of idle/busy/just woke from sleep logic
- improved how the controller maintains the db and memory
- improved force_idle mode in the client
- idle calculation is more accurate, and resets on more user-driven events
- improved responsivity for processing popup
- fixed a serious bug that was counting some unprocessed content as done when the service processing window was cancelled
- misc code improvements
- misc maintenance code improvements
- fixed a bug with parentless center-positioned dialogs not sizing themselves properly
- removed delete_orphans timespan param option
- fixed bugs selecting and media-viewering collected media
- fixed a missing thumbnail selection bug
- fixed some os x dialog interaction and z-order problems when the splash screen is open
- os x splash screen is no longer stay_on_top
- fixed the 110.00% display bug
- removed wx from pubsub code
- pubsubs now route through the controller
- reduced some pubsub overhead, we'll see if it smooths out some of the gritty stuff
- misc thread worker pool is now managed by and routed through controller
- some other thread-wx interaction is now routed through clientcontroller
- db now interacts with controller directly, not through wx
- all other indirect references to controller go through hydrusglobals, not wx
- refactored everything wx away from everything hydrus/server except controller code
- about window now says the client's network version
next week
I also did a lot of code cleaning this week. The server is almost completely divorced from wx–I just need to refactor the controller object, and then I can make it properly headless, so that is my first priority. After that, I have a lot of small things to do.
Version 172
The video is long, but it was absolutely my favourite thing all week. If you have some time, get comfy and watch it in HD.
win32
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v172/Hydrus.Network.172.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v172.tar.gz
I had a good week. I've made the backbone of both the client and server work better, and I've done a bunch of small other things.
headless server and better controller
The server is now completely headless. If you run from source, it doesn't need wx, and it now wants to run from the command line/terminal. This is not complete, and the more complicated commands do not work yet. Just run 'server help' from the command line for more info. Basically, for now, run the server as usual, and when you want to quit, hit ctrl+c a few times. I will add support for external stop/restart and better command line feedback next week.
Also, server.pyw is now appropriately server.py, so update your shortcuts if needed.
As part of this, I have rewritten quite a bit of the client and server startup and shutdown code. Everything tests ok my end, but please let me know if your client hangs on shutdown or anything!
Also, I rewrote the code that detects if the client or server is already running to use a slightly different test that should be a lot more reliable, working even if different instances were launched using completely different means, and hopefully immune to false positives.
couple of new options
You can set the default 'file limit' for a gallery page in the new file->options->downloading page. This applies to new subs as well.
I have also added an autocomplete embed option to file->options->gui, which lets you force the autocomplete dropdown to always embed just below the text entry box, like it does in the manage tags dialog, rather than being a floating window. This is now default behaviour for Linux and OS X, which often have key and focus event processing problems with the complicated floating frame my dropdown creates.
Also, the manage services dialog lets you reset repositories more thoroughly, letting you force a re-download of all updates (this is useful if you were screwing around in the update directory and accidentally deleted something, or if those files otherwise somehow become corrupt). The reset buttons have better workflow as well, and report their (often slow) progress in a popup message.
full list
- the server no longer needs wx!
- the server now runs from command line
- the server is now .py, not .pyw!
- run 'server help' for more info
- improved how the controllers start
- improved how the controllers stop
- improved controller boot error handling
- improved how pages close
- general controller code cleanup
- general server code cleanup
- fixed the always_on_shutdown (without asking) option, which was asking anyway
- removed a debug statement in isalreadyrunning code, whoops!
- improved isalreadyrunning detection. it should work breddy gud now
- added gallery file limit to new 'downloading' options page, which folds in the old thread checker options as well
- added option to always embed the autocomplete dropdown window (rather than having it a floating window), which is now default on for Linux and OS X
- manage services now supports two kinds of service reset for repositories
- the service reset buttons now only fire on dialog ok
- service reset will try, as cpu allows, to update its progress in a message popup window
- administrators now have a 'sync now' button on the review services window that lets you catch up immediately to the service without having to wait for the normal update time (this will burn cpu time serverside, so be careful!)
- fixed a bug when searching boorus with unicode-16 characters
- the client updates directory is neater
- 'system busy' status is now shown on the status bar
- 'force unbusy' added to debug menu
- invalid characters in export filenames will now be replaced by underscores
- fixed a bug where rating services' cached file counts were not decrementing on de-ratings
- rating services' cached file counts are reset on update
- hover windows will pop in over video again, but will not if the mouse is _near_ the animation bar
- searching for ':' in the autocomplete dropdown will no longer search the db for every single tag jej
- improved some server_busy logic
- improved some server shutdown logic
next week
There is some more server stuff to do, and I want to update the import folder code, including adding some more options like letting you designate a place for failed files to go. If I can get both of those things done, then I am ready to start the big job of switching subs and gallery downloads over to the new downloader engine.
Clearing all tags on selected files
How do I remove all the tags on files I have selected? I did a bad import and now I have a ton of tags on my files that I don't want, and I can't figure out a way to get rid of them in bulk. Is the only way to clear a tag from a file to go into the tag edit mode and double-click on the tag? Because that gets really cumbersome, considering you can't select multiple tags at once that way.
I've been working on getting tags -> exif working with a friend.
Uses a powershell script to regex tags out of a filename.
I've brought the idea up before, and we know it's not for putting files back into the database due to a changed hash, but it's more so for external storage.
Back on point, I seem to have a compromise on our method, as we overlooked the maximum amount of characters allowed for files (I defaulted to this case as it's a general IO error, exhibit a).
A new method I'm concepting, is to match a image hash, example 5a1.jpg to a palintext document, example 5a1.txt.
From there, as we've already regexed' the filename, we can query a text document to scan an use a program to add these tags to a files exif.
Problem is that since a while ago, tags used to be stored differently, allowing them to be obtained using regular methods, but this was scrapped in favour of a better system for databases.
So I'm asking if it's able to implement a method that allows plaintext tags, and if possible in a method that doesn't seem counter intuitive to our current method
Currently, we've been using Exhibit B to export our files
Release Tomorrow!
I had a good week. I cleaned a lot of code, including making the server simpler to run, fixed a bunch of bugs and generally worked on some quality of life stuff, and added some small things.
The release should be as normal tomorrow.
uploading bug
Hey hydrus dev, got a bug, not sure whats wrong
have someone else hosting the repo itself, but desu image repo is up, people are able to download from it, test it in manage services and it gives an "everything is working"
but, when i try to upload images to the repository, it times out on me, i have 169 images waiting to be uploaded but every time i hit upload, it just times out on me, any ideas?
force client to always use a server ?
Hello, I have set a server, and I would like that every actions on a client are done on the server.
Is the local server mandatory ? when I do a download, can it be done on the server directly ?
How exactly do I install on Linux?
The install instructions seem exclusive to Windows.
Release Tomorrow!
I had a great week. I fixed some bugs and cleaned some things, and I rewrote the way db maintenance and repository processing are scheduled so they are much less likely to interrupt you. There are some new options to further customise this behaviour. Furthermore, I sped up repository processing so it is at least twice as fast.
I have some IRL stuff going on, so the release tomorrow may be a couple hours later than normal.
Version 170
win32
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v170/Hydrus.Network.170.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v170.tar.gz
I had a great week. I did several different things. If you have a lot of files, it will take a few minutes to update.
media viewer options
In options->media, you can now tell the media viewer how to act for each supported mimetype! You can set it to show media as normal, or start animations in a paused state, hide anything with an embed button, just show an 'open externally' button, or even just open externally straight from the thumbnail.
This was a long time coming, but I am really pleased with it now. It should make managing weirder media, particularly on non-Windows, a bit easier.
To harmonise things a bit across platforms, I have removed the weird Win32 audio/media plugin that didn't work well–audio is now importable but not internally renderable in the client. You'll just get an 'open externally' button. FLV will render in the native video player as well, rather than the flash plugin. Flash will still render in Win32, just because that plugin works so well.
The media viewer is now ready for me to plug in a native audio renderer, which I hope to write fairly soon.
misc notable stuff
The badly named 'url' download page is now 'page of images'. I have tweaked its parsing rules as well, so you can better choose which kinds of images it will go for. It is moved to the new downloader engine, so it has the new import status window button and will remember its status through session restarts.
Right clicking on a tab lets you rename it! Renamed tags will persist through a session restart!
The deviant art parser will now parse mature images.
In options->files and trash, you can tell the client to remove files from view once you filter them.
The UPnP dialog has better error management. Also, if your client has problems figuring out your external IP, you can override it with whatever host you like at options->connection.
Deleted files should now go to your OS's version of a recycle bin!
os x release
To improve https support on OS X this week, I rebuilt my OS X development environment. It was a headache, but everything seems to be working ok. Please let me know if something that used to work suddenly doesn't! I am particularly interested if there are any 'ssl' errors still hanging around.
full list
- added media viewer mime options to file->options->media page
- you can now set animated mimes to start in a paused state
- you can now set the client to 'open externally' any media from a thumbnail activation
- reworked the way the media viewer generalises and displays its media to obey these new options
- improved some misc zoom code
- flv video is now rendered natively by default
- audio now defaults to an 'open externally' button
- renamed the nebulous 'url' download page option to 'page of images'
- moved the 'page of images' download page to the new system–it'll now remember its state through a restart, has a the detailed import status button, all that stuff
- 'page of images' download page now supports checkboxes to customise which types of links it will search for
- you can now close and rename page tabs from a right-click menu!
- deviant art parser now works for mature images
- improved deviant art tag and image page parsing
- added 'remove filtered files' option to file->options->files and trash
- improved UPnP dialog error reporting for when external IP cannot be parsed and for when external IP is reported as 0.0.0.0
- added an external ip/host override option to file->options->connection page
- whenever the client or server deletes a regular file or directory, it will now send it to the recycle bin
- files stored in the db are no longer read-only
- existing files stored in the db will attempt to be set read&write on update
- wrapped undo manager access in a lock, which may stop index bugs when a lot of stuff is going on
- fixed a bug in download progress->progress gauge hook when content length was absent
- removed previously entered zero-length namespaced tags (like 'wallpaper:') from client and server
- moved 'client/server already running' check from db access test to more reliable process list review
- improved 'client is already running' mini-dialog
- added cpu percent utilisation check (if any cores >50%, dump out) to idle check
- if the 'canvas zoom' value exactly matches a default zoom, it will not be inserted into the list of zooms to scroll through
- the 'canvas zoom' will be displayed with hundreds of a percent accuracy
- the new thread and hdd import pages will now obey page-global pause/resume events
- the new thread and hdd import pages now have the 'sort by' dropdown
- the new thread import page supports set_search_focus event
- added import tag options info to 'getting started - tags' help page
- relabelled the path tagging button on the import files dialog to something that made more sense
- might have patched an unusual subscription error reporting bug
- updated sqlite on windows–db should be a bit faster
- updated pillow on all platforms
- compiled a new python for my os x build with newer openssl library, which should fix some ssl problems people were having
- updated some deprecated directory parsing code
- general code cleanup
next week
I have some good ideas on how to improve the cpu hit of repository processing, mostly by adding user options governing how it occurs, so you can change it to work how is most convenient to you. Otherwise, I want to do quite a bit of code rewriting.
I had a great week. I added per-mime view-handling options for the media viewer, updated the generic gallery downloader to the new downloader system, and fixed a ton of bugs.
I have some OS X build environment things to test as I put the release together. If that all goes well, the release should be as normal tomorrow.
What does "clear deleted record" mean?
Pic related
Local tags->Repositry Request
I've been using this fine software for a time, and I love it. It made my life easier.
I tag very much, but due to my slow hard drive, I don't upload them to repository. (I plan to get a decent pc with SSD though)
My "simple" request is, that the client be able to copy all the local tags of the selected files, and paste them individually to the repository tags.
Local/Repository sync feature that is.
I know that it's not as easy as it sounds (I know how hard programming can be), but I do know that the users could contribute more to the tag repository system this way.
Version 168
win32
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v168/Hydrus.Network.168.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v168.tar.gz
I had a good week, spending most of my time updating the thread watcher to new code.
thread watcher
I have converted the thread watcher to my new downloader engine. Beyond some general improvements to its interaction logic, it will remember its state throughout a session restart and it now has the new button to show the status of every import item, so you can retry any failed downloads.
The gui layout is new as well. I've tried to compact it, only showing you useful information.
The thread watcher now interprets the 'times to check' as total times to check, rather than times to check after the first time, so you might want to bump the default value up by one in the options, particularly if you have it currently set to 0!
This rewrite took about twice as much work as I expected. Subscriptions and gallery downloads are significantly more complicated, so while I would like to get them equally up to date as soon as possible, I may have to try splitting their updates into smaller jobs.
new icon buttons
You can now click a button to 'open externally' in the media viewer's top-middle hover window, and there are separate buttons for 'send to trash', 'delete permanently from trash', and 'undelete'.
full list
- thread watcher is moved to the new system–it will remember its previous state
- thread watcher has more compact, flexible gui layout
- thread watcher now supports file import status button, so failures can be reattempted
- thread watcher check now button has improved logic
- import tag options and import file options gui controls now plug into the new system
- thread watcher will now remember its tag and file options
- thread watcher will accept changes to its tag and file options after it is started
- fixed a bug in import tag options deserialisation
- hdd and thread watcher static texts will flicker less
- improved some network transfer gui reporting code
- send to trash, delete from trash, and undelete now all have different icons in the command hover window
- added open externally button to command hover window
- opening an animated gif or movie externally from the media viewer will pause it in the client
- share->copy->image added to media viewer for static images
- fixed a typo bug in copy 'image' to clipboard
- fixed a typo in external ip discovery that was causing errors for local booru external link generation
- improved external ip discovery error recovery
- in prep for the de-wxing of the server, refactored a lot of code so the server and client code don't import each other at any point
- if an attempt to close the client is aborted (e.g. because of an active import page), the exit splash screen will now destroy itself
next week
My help files are out of date, so I will do a full read-through and update the text and screenshots to what is current. If you are a semi-new user and had trouble understanding something in the help, let me know now and I'll try to improve that particular section specifically.
Other than that, I think I'll look at adding some options for default actions for each mimetype, so you can say 'always open flash files externally' or 'show the embed arrow before you start playing a video' and so on.
Delete files to recycle bin or trash
Can we get the client to delete imported files to the recycle bin on Windows or whatever file trash is in the Linux you use?
Noob Questions
Hey, noob here, trying to figure out how to get tagging to work. I heard that I could use Hydrus to automatically scrape tags from sites and tag based on hashes, but I can't figure out how to do that. Do I have to wait for the repository sync to finish? So far it's taken several days for it to sync and it still isn't 50% done, it seems to get slower and slower as time goes on. It started off at a speedy 200-300 rows/s, but it's down to 12 rows/s, sometimes dropping to 1-2 rows/s.
I have also had issues with Hydrus lock up seemingly at random. Sometimes it'll start responding, other times the process has to be killed and program restarted.
My system is an AMD FX-6300, 12gb ram, R9 270 if that helps diagnose any issues
Version 169
win32
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v169/Hydrus.Network.169.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v169.tar.gz
I had a good week mostly rewriting the help. There are also some bug fixes.
help
Keeping help files up to date is a chore. It takes a long time and there are always more fun things to be working on, so it is easy to put it off.
I put it off for way too long, so nearly everything needed work. New users were often thrown off.
So, I've gone over everything, caught it all up, made it all a bit more concise and explicit, and updated the screenshots. As always, the online link is:
http://hydrusnetwork.github.io/hydrus/
And you get a local copy in the release. I hope this makes it a bit easier for people trying to pick the software up. Let me know if I have forgotten anything. I have made a repeating task in my to-do to regularly recheck and update the help in future, so I shouldn't have to do a full rewrite like this again.
By the way, some users are looking at working on a wikia placeholder I created ages ago and mostly forgot about. If you are interested in writing an article, please check it out here:
http://hydrus.wikia.com/wiki/Hydrus_Network_Wiki
If this turns out to be a place where tutorials or a list of repos ends up or something, I'll link it officially in all my usual places.
full list
- revised all the help files
- updated out of date screenshots in the help files
- added some more screenshots to the help's index
- cleaned out unused help images
- fixed some missing service error catching in the custom filter setup dialog
- fixed random reordering of paths from import files selection dialog
- logged errors should now be prefixed by the time they occured
- errors should now be more reliably printed to the log
- fixed a problem with e621 queries that included '/'
- improved some server temp file error handling
- fixed a typo that was throwing errors when trash settings were set to 'no limit'
next week
I didn't have time to add default mimetype handling options (e.g. by default, open mp4s externally), so I'll do that first. There's also some general rewrite work and other little jobs that I'd like to clean out before I start something big.
Will this be addressed?
Was too lazy to set up a width:x height:x in the query, and did it through the client
Release Tomorrow!
I had a good week. I rewrote all the help and fixed some bugs.
The release should be as normal tomorrow.
Daemon MaintainTrash
I keep getting these error notifications:
"Daemon MaintainTrash encountered an exception:
TypeError
unsupported operand type(s) for *:'NoteType' and 'int' "
I use the extract only version. I bet it has to do something with the trash maintain settings, tried to reset them but didn't work.
Release Tomorrow!
I had a good week, although I had a little less time than usual. I completely converted the thread watcher to the new download engine, so it works a little better, will remember its state through sessions, and also supports the new import list so you can retry failed downloads. I also fixed some misc bugs and added some buttons to the media viewer for open externally and the new delete operations.
The release should be as normal tomorrow.
Image dimensions
Is it currently possible to display the current image dimensions in the status bar at the bottom of the client?
If not, consider this a feature request.
Would be nice to be able to copy the image to the clipboard, rather than just hash/path/etc
howto
Could you please redo the manual for hydrus?
The biggest problem i find with trying to get people on board for hydrus is "I don't know how" and the guide is old and not correct
I have heard "I don't know how" and "its complicated" a lot of times.
If not, maybe everyone here on /hydrus/ should work together to make a new guide?
Version 167
win32
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v167/Hydrus.Network.167.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v167.tar.gz
I had a good, creative week. Almost everything in the changelog is gui related.
drag and drop export
You can now drag and drop any local thumbnails out of the client window, and they'll export! This also works with the 8chan uploading box.
This has been a long time coming, but I got there eventually. It works really well for me, but let me know if you have any problems with it.
new control to review file import status
When you perform a hard drive import, there is now a new button next to the pause button. It launches a window with a big listcontrol showing information on every import item and its current status. It will update as the import proceeds. You can right-click for a menu that'll let you retry all the selected entries.
I will add this new control to the other download pages and the subscription system as I move them to the new downloader engine. It is also easily expandable–I'll probably add a menu entry to copy error information to clipboard and so on.
full list
- created gui control to show file import status
- added a button to launch this control, wrapped in its own frame, to the hdd import page
- you can now drag and drop thumbnails out of the client for quick export!
- the client will catch internal drag and drops, and won't interpret that as an import request
- dragging files onto the client will now always show the 'copy' mouse icon
- some thumbnail generation code is a bit quicker–resized thumbs are now always generated on their first viewing
- if you use 200x200 thumbnails, the client won't bother to generate resized thumbs, saving space and time
- the text entry dialog will now check its button status on a non-keyboard related text-change (e.g. selecting paste from the right-click menu)
- the popup created by downloading a raw url or a youtube video will finish itself properly when the import attempt is completed, removing the pause and cancel buttons and making the popup dismissable
- the splash screen will now stay on top
- the password entry dialog will also stay on top
- you can now copy the md5, sha1, or sha512 hashes of local files from the normal share->copy->hash menu
- fixed an overly broad deleted/redundant test on md5/url pre-import status checks
- options to remember size of manage tags and position of both manage tags and ratings dialogs added to file->options->gui
- updated some of the help
next week
I want to start work on an important rewrite of way the server runs itself, removing all its gui components so it can run better from command line. This will be a big job, so I'll try to do a little bit every week.
I would also like to continue moving download stuff to the new system. I am not sure what'll be best to do next–maybe subscriptions.
"open file location"
can we PLEASE get an "open file location" button for images?
I end up needing to do that a lot and have to jump through hoops to do it
Release Tomorrow!
I had a very good week. I created a new gui control for import pages that shows the status of every import attempt, added slick drag and drop thumbnail export, and improved several other, smaller gui things.
The release should be as normal tomorrow, maybe an hour or two late if I get delayed with some IRL stuff.
open an image in existing copy of Hydrus
Quick question: is it possible, via a command in a Python program, to open an image in the copy of Hydrus that is already running on my computer? Hydrus is not the default image handling app, and even if it was I believe calling it via os.startfile() would still launch a new process of Hydrus.
A way to use all those images we all have
This may be of interest to users of Hydrus: I finally got off my ass and wrote a replacement for the janky pile of garbage slideshow program I used to use to view my export folders on my secondary monitors.
https://github.com/VerstandInvictus/DevilScreens
This was coded because I needed something to cycle through my export folders, but it should work for any collection of images whether or not they came from Hydrus. If you have your export folder filenames set up to include "namespace", then that namespace is displayed as a label at the bottom of each image.
Still sort of rough and under heavy development, but totally usable if you're ok with INI files. Controllable (next/prev/pause/play) and has an "open file" button. Please do submit feature requests/bug reports/pulls/whatever on Git or in this thread.
Why two thumbnails per file?
Each pair of files aren't the same file but they're close to the same size and look like they're the same resolution. I have close to 4 gigs of thumbnails, do we really need half of them?
Version 166
win32
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v166/Hydrus.Network.166.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v166.tar.gz
I had a good week, though I was short on time. Several small things work a bit better.
hdd imports now resumable
For a while now, I have had planned a big rewrite of the downloader engine, which gets files from whatever source and bundles them with tags and so on for the db to import. It is a big job, and it has been difficult trying to break it up into pieces. This week, I converted the hard drive importer (which is a lot simpler than the gallery downloaders, and so best to start with) to the new concept. I am really pleased with the result.
Hard drive import pages will now work quite a bit faster, with less chance for problems, and simpler gui controls, and are also resumable through a session restart. This means, for instance, that if you have hydrus set to start where you last left off (default session set to 'last session' in file->options->gui), and you restart the client half way through a hard drive import, the import page will remember where it was and continue if it was closed in an unpaused state.
I hope to continue this conversion to all the other downloaders (including the subscription system, which shares a lot of the same code) so that gallery downloads will remember their queues, and I also want to create a gui control that will show detailed status for all urls/paths/whatever in the current queue, which will allow reattempts of failed files.
As part of this, I removed .zip support from hydrus, both import and export. The .zip code complicated things and it didn't work very well. I may introduce it in another form if people want it.
repository improvements
Downloading files from a file repository is easier. They should start downloading as soon as you click (before, they had a three second delay due to an oversight), and I have added a little popup to give you some feedback on what is going on.
Server admin is also improved–backing up is now a non-blocking operation, and the server will report to clients with a special error code that it is 'busy' while the backup is proceeding. This stuff is a change to the network protocol, so I should technically increment the network version, but since it is admin stuff only, I didn't. If you run a server, make sure to update it alongside your client this week.
full list
- created new object to hold hdd import information
- created new object to hold generalised import file status
- moved hdd management controller to the new system
- moved hdd management gui to the new system
- hdd imports will now remember their import and pause status through a session change
- misc import code improvements
- hydrus client is getting out of the zip business–zips are no longer parsed for import nor able to be created for export
- import and export code is simpler
- repository file downloading daemon will now throw up a small auto-dismissing popup when it downloads files
- repository file downloading daemon will respond to new downloads much quicker than previously
- if a repsitory is in the process of backing up when it receives a request, it will now return a 503 'server temporarily busy' error (rather than timing out)
- the client can now catch and handle these 503 errors gracefully
- backing up will no longer block the client's gui, and it will no longer timeout if the operation takes more than ten minutes
- some networking code is a little simpler
- added 'select local files' and 'select trash' to thumbnail right click menu
- 'cancel download' added to file repository submenu of thumbnail right click menu
- some buggy listbox resize behaviour (scrollbars not disappearing and occasional layout and drawing update fail) has been fixed
next week
I am not sure. I think I will focus on making the url status gui control as I describe above. Other than that, I might have a look at dragging thumbnails out of the client for quick export/upload.
Release Tomorrow!
I had a good week, though I was quite short on time. I added some gui stuff to help file repository interactions, caught some server errors better, fixed some graphical bugs with tags lists, and I started work on the downloader engine rewrite, making hard drive imports work a lot faster and permitting them to resume after a session restart.
The release should be as normal tomorrow.
Basic question.
If I was querying a booru for downloading, (eg Danbooru)
How do I filter tags on the download
using -{tag} still grabs images that I do not wish to import
Video issues
Hydrus_dev wanted info on how the video player was working. First a question, where are the playback controls?
I'm not getting sound in any videos I've tried.
Seeking the video with the time slider often freezes playback and it won't restart.
Playback appears to be too low framerate in many videos but this is based on perception, there is no time indicator to compare with or audio to listen for sync loss.
There seems to be a memory leak in Hydrus that leads to "memory errors" after about an hour of browsing images, but videos make the error happen much sooner.
Gifs converted to webms seem to work best.
I would really like to have a default option for particular mime types to only show the thumbnail in the browser and open externally if they're clicked.
Worth fixing?
pls to help
I'm recovering data from a hard drive. Which files and folders are essential to keeping all of my database files and tags, if nothing else?
Cheers.
Version 165
win32
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v165/Hydrus.Network.165.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v165.tar.gz
I had a pretty great week. I did a bunch more on my new trash service and fixed a lot of stuff. The update to v165 may take a few minutes.
the new trash service does more
The trash service will now clean itself–if any file has been in the trash for more than 72 hours, it'll be fully deleted, and if the trash totals more than 512MB in size, it'll clear files (deleting oldest files first) until it isn't. You can change these values in the options, including setting no limits. You can also set files to be removed (i.e. the thumbnails disappear) from the current view as you trash them, if you prefer the old behaviour.
Furthermore, the delete key will work for trashed files (to fully delete them), and shift+delete will undelete.
And I fixed and improved a bunch of trash-related code. Undo/redo doesn't work yet, but I need to revisit that code anyway.
animations have much better timing
If your CPU can keep up, animations will now draw their frames with much more accurate timing. You may have noticed that gifs and webs were often running 20-60% slower than they should–I thought this was due to a bottleneck in my pipeline, but it turned out it was due to my code making an assumption about something I have no control over being quick. It turns out it isn't as quick as I thought, so now my code will compensate.
I'm really pleased with this simple but important change. >15fps animations look great now.
If your CPU can't keep up (e.g. you are rendering a 1080p webm at full size), the renderer should gracefully slow down.
some downloaders are improved
The 8chan thread watcher now works for multiple file posts! Let me know if you discover any problems with it.
I also added yande.re to the default booru list and improved Moebooru parsing to detect and prefer PNG 'original file' download links if they are available.
some file repository problems are fixed
The trash code from last week screwed up some local/remote thumbnail calculation, which meant looking at a file repository view meant a lot of stupid errors. I've fixed these and some edge cases like knowing to remove the inbox icon when deleting from trash while in a file repo view, and cleaned up a tiny bit of file repo download code. I will do more on this next week.
full list
- added a db table to track when a file was sent to trash
- added an option for whether trashed files are removed from view
- added a max age to trashed files
- added a max size to the trash
- added a daemon to maintain the trash
- improved some generic deleted shortcut logic, so delete key will present option to delete from trash if appropriate
- shift+delete will now undelete
- misc trash code improvements
- thumbnails deleted from the trash will accurately remove their inbox status icons
- images downloaded from file repositories will accurately add inbox status icons
- reduced chance of problems when trying to delete media from the trash when it is currently being rendered
- further reduced this chance
- removed redundant undelete code, folded it into add/delete code
- the media viewer will now remove files deleted from the trash in all cases, even when launched from a file repository search
- significantly improved how animations figure out when to put frames on screen. these timings should be much more accurate, slowing down only if your CPU can't keep up
- 8chan thread watcher now parses all files from multiple file posts
- if a booru provides a link starting with 'Download PNG', which refers to the original source png, that will be preferred over the jpg (konachan and yande.re, which run Moebooru, do this)
- booru parser is now a little more liberal in parsing for the image link
- yande.re booru support is added by default
- fixed some local/remote state code, which was breaking file repository searches in several ways
- improved error handling in repository file download daemon
- cleaned up manage options dialog code
- reduced min size of the media caches in manage options dialog
- moved thumbnail size to 'maintenance and memory'
- added better error handling to repositories that cannot find an update file during processing
- repositories that have such errors will pause to give the user a chance to figure out a solution
- misc code improvements
- fixed a bug where file repository downloads were not importing if the file had once been deleted
- dropped the clever indices from last week–sqlite had too much trouble with them. I will reform my db tables next week to get around the whole issue. for now, I have reintroduced the bulky index and forced sqlite to use it where appropriate
- added a test for collecting pending tag data
- tags in the form 'text:', which would sometimes slip through when typing quickly, are now not processed
- improved tag cleaning error reporting
- improved when special wildcard and namespace predicates are provided
- namespace predicates now display as 'namespace:*anything*'
- fixed a bug when launching the media viewer from collected results
- fixed a command hover window layout bug that was putting namespace text in incorrect places
- fixed a bug that was causing the new client-wide processing phase option not to work because, ironically, its calculation was out of phase
- review services will now state how many updates have been processed
- review services will specify if an update is imminently due, rather than saying the repo is fully synched
- fixed a review services layout bug that was misaligning text after an account or data usage update
- fixed a bug in system:similar_to serialisation, so pages with this predicate can now be saved to a session
- fixed the same bug for system:hash
- vacuum should be a bit politer about waiting for its popup message to appear
- database maintenance won't run while a repository is synchronising
next week
I haven't really touched the file repository code in a while, nor repository administration code in general, so I want to do some of that. I would like to improve user feedback on file repo file downloading (it'll probably be a popup) and add a 'busy' state for the repo that'll let clients fail more gracefully (e.g. "Server busy, please try again later") when the server it tied up with a big backup job or whatever. And I'd like to add a 'sync now' button for admins.
Release Tomorrow!
I had a great week. I added some options for the new trash service, improved some booru parsing, added multiple-file-post parsing to the 8chan thread checker, and fixed a lot of bugs.
The release should be as normal tomorrow.
Version 164
win32
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v164/Hydrus.Network.164.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v164.tar.gz
I had a good week creating a prototype service to handle deleted files. It will likely take a minute or two to update to v164.
trash service
The hydrus client now has a sort of recycle bin. When you delete files from your local file service, they will now no longer be removed–they will instead go to the 'trash' service and get a little trash can icon. Trash files can be undeleted back into the local file service or permanently deleted, which will remove them and immediately delete their physical files and thumbnails from your hard drive.
Once sent to the trash, files will no longer appear in 'local files' searches, but they can found in 'trash' searches. Trash searches work like any other search–you can use system predicates or normal tags, and counts for number of files in inbox and so on will be accurate (inbox status is preserved through a visit to the trash). The trash service also appears in services->review services.
This was a surprising amount of work, but I am pleased. The way file deletion was handled before was pretty bad and did not permit adding an undelete concept. I still have work to do–I want to add a maintenance routine that will automatically delete from the trash after a file has been in it for x hours or if total size exceeds x MB. I also want to add options for default behaviour on normal delete, so people can have deleted files automatically disappear from the current search like before. And I would like to add 'file delete' to my undo/redo system, which can probably do with a complete overhaul in any case. And add select->trash/non-trash thumbnail menu entries.
This trash service is a change to several workflows. Please let me know where it does and doesn't work and if you have any ideas for improvements.
I have removed the 'delete orphans' maintenance routine, as files that are removed from the trash are physically deleted immediately, but I will be reworking the code and bringing it back as a very thorough single-shot database integrity checking function.
boorus
Due to an oversight in the recent gui sessions changes, booru pages were not saving correctly to the database. This meant spammy errors whenever a booru page was open. I have fixed this!
sqlite changes
I have added an index that requires sqlite 3.8.x, which means old sqlite browsers won't load client.db anymore. Hence I have removed SQLiteStudio from the Windows release–there is a newer version of SQLiteStudio, but they reworked the gui, and it went from a compact single-exe program to a sprawling and large directory program, so I can't tuck it into my release anymore.
I have added a new text file to the db folder with a link on where to get SQLiteStudio for yourself if you want it (there's a version for all platforms), and I have included the sqlite command line interface for all platforms. You can do pretty much anything with the command line interface, but the gui is obviously more pleasant.
full list
- rewrote the drawing code for the listbox that displays tags in various ways to be a lot faster and more memory efficient
- updated one new client mapping index that wasn't working quite as I wanted it to something more clever
- db will be a little smaller and mappings stuff will be even faster
- merged the two ratings system predicate input panels, so you can now select like/dislike and numerical ratings system predicates at the same time
- fixed booru download page serialisation, which means they will save to sessions
- prototyped trash service
- locally deleted files will now be sent to trash
- locally deleted files will not be removed from the existing search
- files can be permanently deleted from trash, which will also immediately physically delete them from your hdd
- files can be restored from trash back to the local file service
- inbox state is now more separate from the local file service, so it will be remembered through a visit to the trash
- improved delete code all around
- general inbox/archive db code improvements
- misc content update pipeline improvements
- optimised mass-adding of files to a service (for instance, when (un)deleting a whole bunch of files!)
- delete orphans daemon is removed–it will be replaced by a more thorough single-shot hdd/db purge like 'check file integrity'
- files are not yet automatically removed from the trash–I will add that next week.
- updated db access info in db folder
- added sqlite command line executable to db folder for all platforms
- bit of code cleaning
- cleaned up some gui error reporting
- might have fixed a service cache bug in the db that was causing double bandwidth reports and possible looping sync behaviour
next week
I want to finish off these new options for the trash service, first of all. Then some improvements to the DeviantArt parser to allow mature images, hopefully adding a blogger parser, and fixing several emergency bugs.
I'm thinking about downloading Hydrus.
Is it possible to install it without the server, the web client and the site grabber? I simply want to tag my images, nothing more.
If it isn't possible, I suggest allowing the user to choose which components he wants to install during the initial setup of the program.
What exactly does the "number of words" predicate do?
Also is there a way to search for tags with more/less than x characters in them?
Could anyone try downloading pic related from https://www.8ch.net/cuteboys/res/171657.html?
It's giving me trouble, downloading with the url locks up the client for a couple seconds before skipping over it, and thread watching the page skips it. I'm not sure if I'm a retard and pressed a button I should have somewhere or if it's an actual bug and I should go hunting for logs.
Thanks.
Sorry for NSFW :s
And sorry if there's a proper channel for this kind of thing I can't see any.
push repo updates
so when will we get a button to push repo updates instantly instead of a timed check?
There was a way to make this go faster right?
Finally got the desu repo set up, finally got it updated to current and updateable, now just to test that the images do upload and download, but now were just waiting
the server machines has pushed an image, and my local machine has pushed an image, but neither show up on the other yet, isn't there a way to push updates manually so we can set up the initial repo right away, so if were uploading a large chunk, we can push it, also last time i tried this the server would time out due to the large population of images, but becuase theres no push, you would just have to wait
any way to fix this?
Public tag repo keeps importing update 1129
The public tag repo keeps applying every update since 1129 every time it checks itself. I deleted the update cache files but it's still trying to apply them.
ascii
hey hydrus_dev, just a reminder, you working on ascii art importing?
Would still LOVE to see that made,
thanks
Desu~
e621 tag download
Hi,
I'm currently having some issues downloading tags alongside the images off of e621. Subscription or manual download does not make a difference. E.g. downloading "inbread" should yield two files. A flash and an image. I get tags for the flash, none for the image. Downloading for "littlenapoleon" (which strangely enough finds 34 results via the website and 38 results via hydrus) not even a hand full of files get their tags downloaded.
FYI: I am still on 162, but I didn't see anything in the changelog related to e621 tag download. I also didn't notice this problem when I last used the client a few days ago (not exactly sure when that was).
Cheers.
Chances good for a release tomorrow!
I had a good week that was very full. I worked mostly on my new recycle bin/trash service, which proved to be a slightly bigger job than I expected. I have a bit more work to do tomorrow.
Depending on how that goes, I may delay the release until Thursday. I'll post another update if that happens.
Siblings and Parents for Public Repo
ITT we discuss and propose new siblings and parents.
For the Hydrus public tag repository, there are 37000 tags for Eyes Closed and 8000 for Closed Eyes.
Currently there is no sibling relationship for these tags to my knowledge, so I'd propose we replace Eyes Closed with Closed Eyes (new sibling relationship). It's more of a descriptor that way, although it'd mean replacing more tags.
Any thoughts?
Version 163
win32
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v163a/Hydrus.Network.163a.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v163a.tar.gz
UPDATE: The release I put out last night had a bug that broke tag uploads. I have fixed it and updated the above links. If you got the release yesterday and need to upload tags anywhere, please get the hotfix.
The bug was a single line of code that referred to one of the db indices I changed this week. I assumed that gathering pending tags was in my automatic tests, but it apparently isn't! I'll add some tests next week to make sure this kind of problem can't slip through the cracks again.
I had an ok week improving and fixing things. I didn't have as much time as I wanted, but I'm pleased with what I did.
new indices
I profiled the way the client database was working with mappings and figured out some more efficient indices that seem to do the same work in about half the time. Tag searches, putting results together into thumbnails, and processing tag updates should all be a bit faster.
The update will have to do some heavy CPU work to recalculate these indices, which may take a few minutes.
pixiv
I have updated my pixiv parser to work for their latest code, which means hydrus pixiv pages will now download the largest version of a file again and also parse tags more reliably.
I looked at parsing manga and ugoira pages, and while it is certainly doable, it will be some work. I am about 25% through a rewrite of the downloader engine, so I will probably integrate support for these more complicated types of content as I work on that. I have long considered adding a hydrus-specific 'cbr'-type of file (a single archive content type with multiple internal files), and this may be the time to do it.
rating system predicates
I have reworked the dialog that launches for system:ratings to be much quicker and allow for multiple rating selections in one go. Exact values for numerical rating predicates are fixed for non-'zero allowed' numerical services as well. Have a play with it!
Like/dislike and numerical services are still split into two sub-panels with separate ok buttons, but I will merge them into one next week.
full list
- reconfigured some important mapping indices in the client db to reduce search time for many common tag operations
- the new indices have also sped up tag processing significantly
- added an automatic db optimiser analyzer run whenever the db updates
- pixiv now downloads the largest version of an image again
- pixiv tag parsing improved
- added support for some unusual mp4 types
- the noquery media panel is removed. pages with no query status will now show a normal thumbnail panel with 0 files, rather than 'None' files.
- check file integrity will now report number of missing files and incorrect files separately
- check file integrity can now take a folder to move incorrect files, rather than deleting them
- when you try to pend a tag with a sibling in the manage tags dialog, you will be prompted with the chance to pend the sibling instead
- the rating system predicate dialog now allows for easier (and multiple!) selection of like rating parameters
- the rating system predicate dialog now allows for easier (and multiple!) selection of numerical rating parameters
- the rating system predicate dialog now offers correct predicates for 'zero allowed' numerical rating services
- some bmp icon code cleanup
- changed pause and cancel buttons to bitmap buttons
- cleaned up a lot of pause and cancel code
- now, when popups are paused, they will hide their details
- popups that are cancelled by a dialog change (for instance when subscriptions are changed while subs are processing) will now dismiss themselves after a few seconds
- during repository sync, http failures are recovered from more gracefully, and any pending processing will continue
- the repository update downloading and processing loops will wait on the db and gui more efficiently
- fixed a bug in the way the manage export folders dialog was testing existence of subdirectories
- general code cleanup
- bit more code cleanup
- some good daemon refactoring
next week
First off, I want to prototype my recycle bin service idea and add undoable/redoable local file deletion. Other than that, I have several bugs to fix and other small things to look at.
Version 162
win32
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v162/Hydrus.Network.162.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v162.tar.gz
I had a pretty good week with some disappointing hurdles. In preparation for many future improvements, I rewrote some critical behind the scenes code, and several things you never see now work a lot better. Unfortunately, my changes meant several unexpected things had to be changed, and I was forced to delete some obselete data and also force people to update if they want to keep connecting to my server.
management panels
I have improved how the panel on the left, where you usually put in your search terms, works. It can collapse into a piece of data that stores in the database in a faster and better way, meaning that gui sessions now support all types of page, and they also remember some subtle things like the file and tag services you were searching with and include current/pending status.
With the new system, it will be easy to add new things to remember, like sort and collect status, and exact thumbnail order and selection status. I want sessions to eventually be perfect replicas, including remembering url queues and queue position for download pages. I plan to add these things in the coming months.
Unfortunately, all existing gui sessions will be deleted on the update to v162. The change involves nested serialisation, and I just didn't have the time to write a proper converter from the old system to the new, especially with how bad the old system (which is now deleted) was. The session deletions include the 'last session', if you use that. When you start v162, you will get a single blank page, and an empty list of gui sessions. So, if you want to recreate your sessions, make a list of them and their open pages and their predicates before you update. If you have some thumbnails that cannot be recreated with predicates, like some sitting in an import page, assign those thumbnails a temporary local tag before you update, and then search for that local tag after you update to get them back.
The big rewrite included changing the predicate object, which also affects export folders, which meant I had to update them as well. All existing export folders will be deleted on the update to v162. Do the same thing as for sessions–make a note of all your export folders before you update, and then recreate them afterwards. They work a bit better now, remembering include current and pending and so on just like session pages. Since they are moved to the new system, I will be able to improve them much easier from now on.
Please let me know if you discover a certain predicate or any other search state that will not save to a session. For instance, at one point in testing today, I thought everything was fine and then discovered that ratings and file service predicates were not working, so I had to go back and fix them.
network version increment
While doing this work, I discovered a small flaw in how I was bundling my new network update objects. It wasn't an error, but it was slightly inefficient in speed and bandwidth and quite a bit ugly, and would make analysing serialised objects for problems an additional pain. Since I want to eventually convert everything in the network to my new speedy protocol, and much of the work I have done for management panels uses some of this code, I realised that if I ever wanted to correct the flaw, I would have to do it this week, before I start rolling everything out. This is like getting the foundation of a building level–best done before you have many stories built.
As a result, updating the client or the server this week will take a couple of minutes as some cached data is reformatted to the better protocol.
And because the network protocol has changed, I had to increment the network version, which means that v161 and previous clients will not be able to talk to v162 servers and vice versa. I prefer to give at least a week's warning whenever I do this, but I was forced to make the decision yesterday morning as I saw how very much more difficult it would be to do in any subsequent week. So, if you have a v161 client and connect to my server, you will get a few odd popup errors until you update to v162. If you do not connect to any servers, or if you only connect to servers that you know will stay at v161, then you can update whenever you want.
The pretty 'you are using the wrong version of hydrus for this server' popup message doesn't seem to be working, so I will look into that.
full list
- on update, all previous gui sessions are deleted!
- on update, all export folders are deleted!
- made an important efficiency improvement to the new serialisation protocol–network version is incremented as a result
- finished extracting management data from management panel
- management data is wrapped in management controller, a new JSONable class
- pages collapsed to a single class
- page and management panel instantiation completely rewritten for the new system
- created new JSONable guisession class for better session management
- db is updated to store all this stuff in JSON rather than YAML
- session loading is much less bloaty for clients with many sessions
- all page types are now serialisable and hence addable to a session
- moved predicates to JSON
- moved filesearchcontext to JSON
- autocomplete dropdowns now remember their tag service, include current, include pending and synchronised status over sessions
- export folders now remember their file and tag services, and also include current and pending status
- some general improvements to export folder code
- cleanup of export folder dialog code
- added a test for export folders
- some general refactoring
- some general cleaning
- significant refactoring of predicate variables
- fixed a bug in multi-version update from before v154
- if you do not have any ratings services, the 'manage->tags/ratings' submenu will now just be 'manage tags'
- changed some thumbnail menu entries to say 'selected files' instead of 'all files', which is more accurate, see if you like it
next week
I spent almost all this week on the rewrite, so I didn't have time to get to some important bugs people have reported. I will work on those, and any problems that may appear from all the changes this week, and then have a look at pixiv and ugoira.
Release Tomorrow!
I had an ok week. I improved the way several popups and pause/cancel things work, optimised some important db indices (which seems to have halved the time most tag-related operations take), fixed some problems that developed in pixiv parsing, and made it easier to create ratings system predicates.
The release should be as normal tomorrow.
New system running on my solid.
Should I move hydrus over to it? if so what should I do to maximize transfer efficiency?
Tag siblings not working with multiple files selected
Suppose I have 10 files and one of them has a subordinate sibling tag. It also naturally has the dominant sibling tag.
If I select all 10 files and manage tags, then double click the subordinate sibling and select "add tag to 9 files" then the other 9 files get the subordinate sibling but not the dominant sibling.
Release Tomorrow!
I had a generally good week. I spent almost all my time rewriting an important bit of behind-the-scenes code. It went very well and was an important thing to do, but it ended up making some old data so obselete that I could not upgrade it. I will talk more about it (including how easiest to recreate the deleted stuff) in tomorrow's post, but v162 will increment the network version, delete all existing gui sessions, and delete all existing export folders.
This stuff is important, and with the network protocol change I want to be very careful about my testing tomorrow. Everything looks great so far, but I want to do it all again anyway. If everything goes well, the release should be as normal tomorrow.
Version 161
win32
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v161/Hydrus.Network.161.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v161.tar.gz
I had a great week. I have increased video support for all platforms and fixed a lot of bugs.
animation
I have, for the most part, fixed animation on Linux and OS X. I could not get all gif transparency working, so some gifs with unusual palettes will render strangely, but otherwise, Linux and OS X should render animations and videos just like Windows. In my testing, they generally seem to work great, and I am very pleased. I have updated the static FFMPEG build that comes with the releases, and I added it for the Linux executable release (I think this was missing, before–it certainly wasn't working correctly, even if the user put it in themselves).
I also simplified my video rendering pipeline. It now uses less memory, has less latency, and, it feels to me, renders a little faster.
Furthermore, I have added mp4, wmv, and mkv support to my native rendering system. They now render just like webms, and in the same way do not have audio yet. I plan to add audio in the future.
I would like to know your experiences running these new video types in the client. If you experience crashes, memory errors, slowdown, or anything else, please let me know. There is room to improve my own system, and I would also like to add options for 'always launch mp4s in the default external program' and so on for those people who would prefer to default to MPC or VLC or whatever.
more processing improvements
I have switched the database to a slightly faster transaction mode that should speed things up a bit for all clients and possibly a lot for those with high latency hard drive connections, like laptops or encrypted drives. The drawback is that if your client crashes, there is a very small chance the database will forget the most recent writes to it. Let me know if you experience any significant changes in speed.
I also worked more on the processing code–it should accelerate/decelerate processing speed more quickly as needed and report rows/s more accurately.
BTW: In my testing these past weeks with rows/s, I have found that the easiest way to speed things up is just to keep my drive defragged. The database files can easily split into hundreds of pieces after a big operation, and leaving it like that just slows everything down. I use O&O Defrag and now have it automatically do a quick stealth defrag every twelve hours on all drives, which seems to keep things fairly smooth. It is another one of those things where five minutes a day causes less headache than an hour a month.
a/c dropdown and hover window fixes
I have fixed several bugs with the autocomplete dropdown–it should position and show/hide itself more accurately in Linux and OS X, and the bug that was stopping selecting system predicates with the enter key is fixed.
I cannot get the arrow keys (for selecting tag results) to fire on OS X because of an OS X-specific shortcut-binding behaviour I never knew about before. My long-term plan to improve shortcuts client-wide should eventually fix this, but for now, using ctrl+ or alt+arrow keys seems to work.
Also hover windows should position themselves a bit better for Linux and OS X. I still get some buggy behaviour in Linux, where the windows will sometimes not layout correctly or just decide to reposition in the top-left of the screen, but it happens less now. I will keep working on this.
full list
- updated windows python, sqlite and ffmpeg
- added linux ffmpeg binary to the executable build
- updated os x ffmpeg binary
- animation is generally working for linux and os x
- fixed some linux/os x ffmpeg calls
- fixed webm import for linux/os x
- webms work great
- moved to rendering gifs with PIL by default on linux and os x, which has bad palette support but renders more reliably
- the animation scanbar position caret is now visible on linux and os x
- fixed some timing/resume issues the animation scanbar on os x
- the old mediactrl video embed is removed
- mp4, wmv, mkv are added to native rendering control, no audio yet (just like webm)
- flv and flash will now have a little vertical padding on max zoom to make vertical hover windows easier to access
- changed database to a faster synchronisation mode
- tuned update processing daemon for more accurate time calculations and faster baseline speed, even if that knocks gui latency a bit
- fixed rows/s being thrown wildly off by long pauses
- made update processing daemon more reactive
- improved some of the flow and generally cleaned the update processing daemon
- added processing phase option to regular delay to processing of updates (this is useful if you run multiple clients and don't want them to process at the same time)
- fixed e621 tag parsing
- completely rewrote the focus engine behind the autocomplete dropdown
- fixed a bunch of autocomplete dropdown's generally buggy behaviour for linux and os x
- fixed autocomplete dropdown's display, sizing, and positioning for linux and os x
- fixed autocomplete dropdown selecting system predicates with the keyboard
- fixed autocomplete dropdown show status in linux with multiple pages open
- improved all subprocess calls, removing interim shell step and parameterising passed arguments
- improved how files and directories are launched in windows
- upnp is fixed for linux
- fixed hover window archive/inbox/delete icon buttons, which were spamming their commands to all open media viewers, not just their own
- added a simple raised border to hover windows to better delineate them from the canvas background
- updated the hover window size and position code to be a bit more reliable (still seems to bug out a bit on linux)
- tags hover window should now expand sideways more reliably
- it is now not possible to create nested export folders
- searching for numtags < x will now include files with no tags
- fixed manage boorus dialog OK in linux
- fixed booru selection mini-dialog in linux
- generally improved restoring database code
- fixed restoring a database when you have tag archives in existing database
- fixed a unicode encoding error when converting certain jobs to text (this was throwing errors in deviant art downloads/subscriptions–we'll see what was actually going on now, whether this is an error not being formatted right or something else)
- started some great rewrite of management panel gui code
- rearranged the download panel gui hierarchy slightly
- improved collapsible panel collapse/expand layout code
- removed code and database table for the old numerical ratings filter
- some general code refactoring and cleanup
- improved client upnp daemon timing
- harmonised how the client and server check requests for bandwidth-tracking eligibility
next week
I am on the cusp of committing myself to a severe and sorely needed overhaul of how the client's pages work behind the scenes, so I will start that first thing to maximise the chance that I will finish it in time for next Wednesday. I want to eventually have 100% accurate session support so that import pages will be resumable, and thumbnails will remember if they were selected, and so on, which means revamping some some integral stuff.
If I have time after that is done, I have some pixiv and danbooru parsing to look at and potentially adding native ugoira support.
I started reorganising my to-do list this week from a bunch of overloaded colour-coded Zhorn Software Stickies to a richer AbstractSpoon ToDoList Tasklist, and I am really pleased with the shift. I felt that some feature requests and bug reports were getting lost in the pile, so as I process my old lists of thoughts into proper actionable jobs, I hope I will be decreasingly able to lose track of them. If I have forgotten something you asked for a while ago and it is important to you, please tell me again and I will bump it up my priority list.
pImgDB
Hey hydrus dev, this seems to be a sort of precursor to your program, im curious if you might actually have it
Running from source
I love Hydrus, and have at several points tried to do some modifications to the source code, to add some functionality or maybe refactor some code.
A problem I always end up is to actually run my home-brewed version, as I am always missing some dependency or other.
Thus, I would like to make a thread about getting this thing to work, directly from source.
I encourage you all to join in, if you have a problem or know the solution to one. Hopefully hydrus_dev will also help now and then.
Some basic, but inadequate info can be found here: https://hydrusnetwork.github.io/hydrus/help/running_from_source.html
To start off:
Right now I am incapable of importing the function RAND_egd from the ssl module. I did some searching, and it seems to gather entropy for use in secure random info. At runtime it seems to require a special file for this, but that doesn't explain why it won't import. I am using windows.
Release Tomorrow!
I had a great week. I have expanded my native video rendering to support mp4/wmv/mkv, got video and gifs to work for Linux and OS X, fixed several issues with the autocomplete dropdown for all platforms, improved some important internal data-handling, generally cleaned code, and fixed a lot of small bugs.
There will be a lot to test, but the release should be as normal tomorrow.
Saving custom regex
Can we get a way to save custom regex for when we import? I have to keep a text file full of them and copy and paste 4 or 5 of them into the import window every time.
Version 160
win32
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v160/Hydrus.Network.160.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v160.tar.gz
I had a pretty good week. I was a little short on time, but I added some neat things and fixed some important bugs.
proxy prototype
A user requested proxy support a little while ago because their ISP was censoring 4chan. I have written a very basic prototype for http, socks4, and socks5 support–look in the normal options dialog for the new 'connection' page.
I used free proxies to test this, as I don't have one of my own, and I found it to be very buggy. Sometimes it would work fine, sometimes it would say the 'remote host refused the connection'. This could be the free proxies being choked and just turning down some requests and me not handling the error properly, or it may be that I coded this wrong. My networking engine is a bit hacked together, and I would like to slowly improve it over the coming months, which as well as improving reliability for proxying will allow for better error management, resource usage, and the ability to fine-tune proxying to only work for certain types of request–at the moment, proxying is client-wide.
So, if you have need for proxying, please try this out and then report what sort of experience you have, be that no problems at all or only errors.
custom ratings colours and shapes
You can now set some new options for your ratings. Check them out in services->manage services.
Other than the obvious colours and shapes, you can set numerical ratings to not allow a 'zero' rating. This means, for instance, that you can't set 0/5 for a five star rating, only 1/5 through 5/5. This slightly changes the mouse-click behaviour of the control.
If you have a lot of existing ratings for a numerical service and want to change its 'allow zero' value, make sure you increment or decrement the number of stars so that the number of possible ratings to assign to a file remains the same, otherwise your existing ratings might get changed. e.g. If you have a 4-star rating that allows zeros, there are five possible ratings, from 0 to 4. If you change it to not allow zero ratings, bump it up to a 5-star rating at the same time so there are still five possible ratings, from 1 to 5. If you don't understand this, I can explain it more, but for now don't touch 'zero allowed' if you have a lot of rated files.
quick autocomplete entry
You can now type something real quick into the tag autocomplete entries and hit enter and it will go straight through! You don't have to wait for the db to fetch autocomplete results–if you hit enter and the results are not yet loaded, whatever was in the text box will go through. I hope this makes searching and tagging a lot easier!
The siblings manager steps in as well, so if you type in 'mmd' and hit enter, you'll get mikumikudance.
rows/s
I screwed up the rows/s calculation last week. The numbers were ok, but due to a typo the average wasn't being updated correctly. You may have been scratching your head at something like 12 rows/s. I have fixed this and further improved the accuracy of the initial time calculation, and I have changed the update popup to show 'content rows' rather than 'content parts', so the rows/s should mean something more useful.
At peak, in idle mode, my aging dev computer typically gets 1,300 rows/s. I'm pleased with that number, but I've done a little more profiling, and I think I can double it in idle mode.
full list
- added options for http, socks4, socks5 proxying
- improved some network-related errors
- ratings services can have custom border and fill colours for their various states
- ratings services can now also be squares or stars instead of circles
- numerical ratings services can disallow zero ratings
- fixed the JSON parsing error that broke the thread checker
- shrank the width of the thread checker to make it a little less ugly
- the autocomplete tag search dropdowns will now accept and search with quickly entered text
- this new system will substitute siblings in the manage tags dialog
- fixed rows/s average calculation
- rows/s is more accurate
- content update popup string update is less laggy
- content update popup now shows content rows, rather than content parts
- removed update 'taking a break' component, as it was not doing the job I wanted it to
- db debug profile mode can now be turned off lol
- fixed an error from middle-clicking greyspace in the linux notebook tab area
- general code cleanup
- some string conversion code cleanup
- fixed a missing canvas bmp error with flash/flv embed buttons
next week
I want to sort out animations on Linux and OS X, which I have put off for too long. Beyond that, a bit of code cleaning and generally working my list. I also have a lot of github issues that I want to put some time into.
Public tag repo siblings for Japanese names
Boorus tend to use romaji names for artists. If we start submitting siblings for Pixiv artists and their Japanese names, would you prefer the dominant sibling to be the Japanese or romaji?
I'd prefer romaji just because I don't want to search tags with japanese characters.
Version 159
win32
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v159/Hydrus.Network.159.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v159.tar.gz
I had a very good couple of weeks. In case you missed it, a giant update was causing clients to crash from memory overload, so I shut down my server and put off last week's update to get a good fix done.
So, I have completely rewritten the update network protocol for hydrus, including how updates are generated serverside and stored to disk, to keep memory and cpu costs low. In my general testing, memory usage stays below 50MB (before, the problem 1.1 million tag update attempted to reserve many GB of memory and hence was crashing the client), and converting an update between a network string and a useful object is about two hundred times faster. Even better, I have saved about 30% on bandwidth.
Because of this important change, this release increments the hydrus network version, which means that v158 clients cannot talk to v159 servers and vice versa. I am turning my server back on today, and if you would like to keep syncing, you will have to update your client.
Furthermore, your client will have to redownload every update for every repository it syncs with. It will not have to reprocess those updates, only download them. For my public tag repo, it'll be about 220MB.
update rewrite
Essentially, I have done two things: I have changed my serialising library from YAML to JSON, and I have split large updates into smaller pieces.
YAML is a great protocol, and PyYAML is a great library that supports it, but it was not working for my purposes. It was really quite slow at doing what I wanted, and it did not allocate memory very efficiently as it parsed. I have scrapped it for the new JSON stuff I have been slowly moving to, which is reliably extremely fast and lean, and my new system allows for easier upgrading in future.
Very large updates are now split into blobs of about 100,000 'rows', where a row usually means something like a tag-file mapping. These sub-updates peak at anything from 1MB to 4MB, which is generally much easier and less laggy to deal with en masse than a giant 15MB+ chunk.
I am very glad I took the extra time to get this right. I would never have been able to get it out on Wednesday in any proper state.
I do not know how many people use hydrus, but I am obviously expecting a bit of a bandwidth rush with all the people who will be resyncing over the next week. If you encounter server errors, please pause your repo sync for a bit and try again later, and let me know what error text or delays or other experiences you have!
Furthermore, I have added a rows/s (i.e. rows per second) report to the normal update processing popup. I tend to get about 250 normally, and 1,000 or so on idle mode. I would be really interested in knowing what other values people get for this. I am also interested in knowing whether you are on a laptop or a pc, and whether it is a particularly powerful machine, whether you are on HDD or SDD, and whether the drive is encrypted or not.
tag export and import
I have ditched the old tag export and import YAML stuff for my new tag archives system.
If you want to export a bunch of tags to another client, just right click some thumbnails and select share->export->tags. This now creates a tag archive, which is just an efficient database file for storing tags.
You can simply import such a tag archive from the main gui's file menu.
4chan dumping disabled
I haven't touched my dumping system in a very long time. Some people told me it wasn't working, as 4chan's captcha has changed or something, and as I don't have time to fix it, I have disabled it. I would like to bring it back with better support for other imageboards.
idle mode
Usually after about twenty or thirty minutes of you not doing anything, the client turns on 'idle mode', which tells update processing to work in superspeed at the expense of making the gui laggy. You can now see when you are in idle mode from the status bar, and you can force it from the help->debug menu.
full list
- split previously monolithic repository updates into smaller pieces
- added service_update server calls
- extended content_update server calls to support sub-indices
- sped up some content update preprocessing code
- improved some content update preprocessing code
- radically reduced serverside memory usage while generating updates
- added iterator splitters to make sure any single update row cannot be too large
- thanks to iterator splitter, updates should process through the client more smoothly
- added timespan splitter to make sure any single server update query cannot be too large
- content updates are resumable if broken part way through downloading the list of them
- the update popup will state how fast it is currently working in rows per second committed
- cleanup of a lot of update related code
- more cleanup of update related code
- improved serialisable protocol so it'll work better over a network
- made serialisable protocol much simpler
- fixed numerical rating system predicate dialog slider range
- fixed numerical rating system predicate dialog OK for valued predicates
- fixed numerical rating custom filter action dialog
- improved some network yaml error handling
- replaced 'export tags' thumbnail menu entry with a tag archives system
- replaced 'import metadata' file menu entry with a tag archives system
- disabled thread dumper and manage imageboards and manage 4chan pass dialogs for now, because dumping code is out of date and completely screwed up
- client will now not start in idle mode
- help debug menu has new 'force idle mode' entry
- idle mode is displayed on status bar
- simplified client main gui status bar display and code
- the flash and flv files' embed button now has a little border to delineate it from the canvas background
- fixed some clientside bandwidth tracking code
- removed some old networking code
- made some custom objects draw themselves more efficiently
- for now, manage tag parents and siblings dialogs will not show deleted rows. I will eventually add a 'show deleted' checkbox, like the manage tags dialog has
- some static image rendering is slightly less laggy, particularly when browsing large images
- fixed initial height of manage tags dialog launched from preview window
- changed the 'search area' vs 'preview' sash gravity so the preview area won't expand or shrink on resize–see if you like it
- autocomplete dropdown should now hide itself when focus goes to other programs
- autocomplete dropdown should now hide itself when focus goes to other hydrus frames in os x
- fleshed out new URLCache object to handle better gallery download url management
- refactored some file status tracking stuff to a better system
- refactored gallery page fetching to a better system
- refactored gallery url handling to a better system
- some redundant import checking should be much faster
- improved error handling when booru image page parsing fails to find image url
- refactored how tags are fetched for DA, tumblr, and giphy as part of the above overhaul
- misc code cleanup
next week
I want to get back to adding Socks proxy support. I could have tried to do it yesterday, but I faced the same dilemma as last week, where I realised that the amount of work it will need means I should not rush it. So, I will do a generic network engine upgrade and then add proxy support, and then just work my list. I want to add those ratings colour options, improve the 8chan thread watcher for multiple files, fix some non-windows issues, continue a multi-month file downloading/importing rewrite, add a recycle bin service, and fifty other things.
With this semi-milestone release, I'd also like to generally say a renewed thank you to everyone for using my program. It is great hearing about the ways you use it and what you would like to see in future, and even though this recent problem took some time to fix, I really enjoyed the work.
Release Tomorrow!
I had a good week, but I was a little short on time. I prototyped proxy support, fixed some bugs, including a serious one that completely broke the thread watcher, and I added custom colours and shapes to the new ratings controls.
The release should be as normal tomorrow.
No release tomorrow!
In case you missed it, my client update code exploded this week due to a very large tag update exposing a bottleneck in my parsing library. I have shut down my public tag repo and started converting my update protocol to a better system.
I feel about 80% done, and I am very pleased with the changes so far, but this is such critical code and such a big network update (once it is done, everyone will have to redownload the entire update cache from every repo they sync with), I do not want to rush it out. I will take some extra time to make sure it works. I will aim to have my release out on Wednesday the 3rd. My repo will stay down until then.
Release Tomorrow!
I had a great couple of weeks. The big conversion of the update code went very well. Update parsing and downloading uses 30% less bandwidth, 95-99% less memory, and 99.5% less CPU. I also fixed a lot of miscellaneous bugs.
Testing has gone great, so the release should be as normal tomorrow. There will be a network version increment, which means updating will be mandatory if you want to connect to an updated server. I will be turning my server back on as well.
Search for files with A OR B?
If I put tags A and B in the filter box it shows only things with A and B.
Is there a way to search for stuff with A *or* B? Is there something I'm not seeing here?
shanachan.org support?
Can we get board support for shanachan.org's /fetish/? The threads there are very active and last for years. Download and posting support would be awesome.
Version 158
win32
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v158/Hydrus.Network.158.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v158.tar.gz
I had a really good and productive week. I had a bit more time than usual, so I loaded up on caffeine and worked my list. In fact, this is the first week in a while where I crossed off more things than I added.
SAVED DOWNLOAD PAGES WILL LIKELY BREAK ON UPDATE! IF YOU HAVE THE CLIENT SET TO RESTORE YOUR LAST SESSION AND HAVE ANY DOWNLOAD PAGES OPEN IN YOUR V157 CLIENT, MAKE SURE YOU ARE DONE WITH THEM BEFORE YOU UPDATE!
subscription tagging
Subscriptions now have a 'get tags even if the file was already in the database' checkbox. It defaults to False–i.e. if a subscription finds a url or hash that was downloaded before (probably in a normal download page a while before), it will skip attempting to re-download both the file and now its tags as well.
Previously, subs were fetching tags for redundant files, which for most purposes just meant wasted time and bandwidth.
numerical ratings control
I have made a custom control for numerical ratings, just like the like/dislike control last week. I have removed the 'lower' option for numerical services–they are now all 'x out of n', where n=5 means five 'stars' and six possible integer ratings from 0-5.
I am not sure I like the colouring behaviour of these–if you use numerical ratings, please have a play with one and let me know if you have a better idea on how to show 'no rating' different to '0/5'.
I will add custom shapes and colours for both like/dislike and numerical ratings in future.
listbooks
Even though I thought I fixed listbooks last week, it turned out there was more to do. I have fixed the display bug when removing a page, and the manage boorus and imageboards dialogs should have less buggy ok buttons.
export folders
You can now set export folders to 'synchronise' a folder, which means the client will make the folder mirror the search you enter exactly, including deleting anything that does not match. Any existing export folders will continue behaving in the 'regular' mode, which just keeps adding.
Also, export folders and the export dialog will attempt to copy file attributes like last accessed and modified time.
exclude deleted
The 'exclude deleted files' option will no longer hide files from file repository results. This option rarely came into play and was nearly always confusing when it did. The option will now just set the default behaviour for file imports–if set, and a pending import file file was already deleted once before, the import will be abandoned.
os x segfault
OS X was sometimes segfaulting and crashing for me on a search refresh. I've added a hacky fix–please let me know if you still get the problem.
preview window menu
You can now right click on the preview window for a simple menu.
full list
-subscriptions now have a 'get tags even if file already in db' checkbox, defaulting to false, which will significantly speed up subs with redundant files
-subscriptions will now more accurately obey initial limit
-gallery downloaders will now more accurately obey file limit
-merged some hentai foundry code into the general downloader
-cleaned and refactored a lot of import page initialisation code
-cleaned and refactored of lot of import management panel code
-cleaned and refactored of lot of import code
-created numerical rating control
-added numerical rating control to rating dialog
-added numerical rating control to rating hover window
-updated like and numerical rating service info for future support of custom colours and shapes
-fixed listbooks page deletion display
-fixed manage boorus saving changes to existing panels
-fixed manage imageboards saving changes to existing panels
-fixed manage imageboards saving remove imageboard
-improved and refactored listbook code, further separating active and proto pages
-improved some listbook name conflict error handling
-misc cleaning
-export folders and the export dialog will also export file attributes like access and modification time
-export folders will now overwrite destination files if they are a different size to the exportee
-export folders can now be set to 'synchronise', and so will delete any other files in the directory. existing export folders will behave exactly as before
-improved misc export and import folder code
-moved folder type tooltip to an explicit bit of text on export and import folder edit dialogs
-thumbnails can now display multiple hydrus file repository icons (current, pending, and petitioned) if multiple states apply
-hydrus file repository icons are gone from media viewer–now there is a text list
-hydrus file repository text list is added to hover window as well.
-exclude_deleted option will no longer hide files from file repository search results–it was more confusing than helpful
-improved some error-prone logic in how advanced_import_options were being stored
-ratings hover window icon background colour is fixed
-imported bmps should now be converted to pngs with 1:1 colour (before, they were being collapsed to a 256-colour palette, which was showing ugly dither on complicated images).
-fixed a sometimes segfault crash on search refresh on OS X
-the preview window now supports a simple right-click menu
-added 'open installation folder' to file menu
-improved some media viewer precaching code, which should speed up some scrolling, particularly on the first scroll
-improved some layout code
-improved some file permissions code
-cleaned some canvas code
next week
I will add some proxy settings to the client so that people who have been hit by recent 4chan/8chan censorship in their countries will be able to go around it via Socks proxying. I don't have much experience in this area, so if you have specific requests on this, please say what you would like to see.
I'd also like to start my recycle bin idea.
Furaffinity scraper
I've made a hash/tag scraper for furaffinity: http://pastebin.com/v7VCZ0W4The database it creates is valid, but I get some strange errors when loading it into Hydrus.
I would love to run it myself, but what with furaffinity's 16*10^6 submissions and my terrible internet connection, it would take way too long.
Shutting my repo down temporarily
The most recent update to my public tag repository is 15MB, or probably something like 1.1 million tags. This is very cool just by itself, but it is also completely overloading my YAML parser and causing MemoryErrors so that the client can never parse it.
As a result, I am shutting down my public tag repository for now so people stop wasting their bandwidth, CPU, and memory. If you don't want to see some popup errors, you can pause all repo sync under services->pause or specifically for my public tag repo in services->manage services. I will attempt to fix this tomorrow with the aim of bringing everything back up for the Wednesday release, but I will keep you updated!
Version 157
win32
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v157/Hydrus.Network.157.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v157.tar.gz
I had a good week. Although I had some IRL stuff going on, I still had some time to work on hydrus. I have fixed the dialog bug and created a new control for 'like/dislike' ratings.
dialog ok bug
At the end of last week, I rushed out a fix to a bug in Linux with my custom Listbook control, which is the list/page 'notebook' that occurs on many dialogs (like manage subscriptions, with the names listed on the left and the data panel changing on the right). Unfortunately, my fix introduced a terrible bug where any pages that had not been viewed (i.e., their name was clicked on) would throw an error on the dialog OK. You probably saw this yourself.
My development/testing environment does not have many pages in, for instance, the subscription dialog, so I happened in my testing of the Linux fix to not hit OK without seeing all the pages, so I never encountered it until my release post was already published.
I am sorry for the incovenience this bug has caused. I decided not to do a hotfix, but perhaps I should have. In any case, the bug is fixed now, and I've added a test to make sure a similar problem does not happen again.
like/dislike control
I have prototyped a control for easy one-click setting of like/dislike ratings. I have ditched the 'wordy' checkboxes for a single coloured circle in the top-right corner of the media viewer. The manage ratings dialog is also converted. I have also added a hover window for these, so you can now set like/dislike ratings without having to open the dialog.
Left click means 'like', right click means 'dislike', and repeating a click on something already liked/disliked will return the ratings to 'no rating'.
This is early code, and I have not converted everything over yet. The manage services dialog still lists the options to set the like and dislike 'words', and there are no options yet for colours–green is like, red is dislike, and white is no rating, with black borders for all. I will add options for colours soon, perhaps including different shapes as well. For now, if you get confused which rating is which, they should produce tooltips.
Since this is so new, I would appreciate feedback. If you find them too small to click on, for instance.
I plan to convert numerical ratings over similarly, essentially having several circles/stars/whatever in a row that will fill up sequentially, just like any normal '3 out of 5 star' rating.
full list
- fixed a bug in listbook page initialisation and retrieval that was affecting many dialogs on OK
- some general dialog OK code cleanup
- fixed a media-navigation bug in managetags dialog
- fixed a serious OK bug in imageboards dialog
- created a new 'periodic' object to manage future subscriptions timing improvements
- started subscription YAML->JSON conversion
- stopped compressing json in the client db so it is human readable and thus easily editable
- subscriptions are no longer renamable, as this code was super buggy and could delete other subs
- tidied up the database menu
- a bit of misc cleanup
- in many cases where it was once pseudorandom, services are now reported in alphabetical order
- prototyped 'like' ratings control
- added new like ratings control to the background bmp of the media viewer
- added new like ratings control to the manage ratings dialog
- added new like ratings control to a new hover window in the top-right
- added basic additional icon support to new hover window
- fixed some misc new alignment bugs related to new ratings stuff
- like ratings controls on the hover window have tooltips
- fixed up some icon/rating display logic in the background bmp of the media viewer
- updated ratings dialog error handling
next week
I am not sure what to do. I think adding some colour like rating options and looking at a custom numerical rating control as I talked about above is a good start. I'm also half way through the YAML->JSON conversion for subscriptions, which is looking great so far and should also improve how normal gallery downloads work.
Release Tomorrow!
I had a really good week. I cleaned code and fixed some lingering bugs, added features to subscriptions and export folders, neatened and expanded some gui elements, and created a new control for numerical ratings.
It will be a larger than normal release, but assuming testing does not take too long, it should be out as normal tomorrow.
Release Tomorrow!
I had a good week. I was short on time, but not as much as I thought. I fixed the important bug that is affecting many dialogs and I have prototyped a new single-click 'like' ratings control.
The release should be as normal tomorrow.
Version 156
win32
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v156/Hydrus.Network.156.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v156.tar.gz
I had a great week. Instead of doing anything particularly new, I fixed and improved existing stuff, including some long-time bugs specific to non-Windows. If you run a server, I recommend you update it this week, as some semi-important error handling is improved server-side.
BTW: I have some IRL stuff next week, so I don't think I'll have time to code and make a release. I'll make a quick post next Tuesday to confirm, but I think v157 will probably be on the 20th.
linux hover windows
I've fixed some Linux stuff this week, but hover window placement is proving difficult to get right. On my Linux dev machine, hover windows (the new info/tag windows I have recently added to the media viewer) position correct the first couple of times the mouse goes over them, and then they jump over to the top-left corner of the screen. If you experience this, or if you don't, I'd love to know!
windows animation fix
Due to a dll conflict in my build process, the Windows version of hydrus forgot how to render certain kinds of gifs a few versions ago. You may have noticed some gifs not quite handling transparency correctly, or having backgrounds with messes of odd coloured pixels. I think I've fixed this, so please let me know if you still get it.
export folder shift
I've decided to better support 'installed' and 'portable' modes. Different operating systems and users have different preferences for this sort of thing, so I want to add more options.
So, from now on, the default 'quick' export folder is no longer install_dir/export, but your_home_directory/hydrus_export. If this changes for you and you don't like it, you can still set it to whatever you like in options.
I think I'll add a dialog on first boot that'll ask whether you want to run the client as installed or portable, and it'll then prompt you to initialise these locations appropriately. I want to maintain the ability to run completely portable, and also add options for other ways of running it. The OS X app build could greatly benefit from this. This is also part of my moving to supporting external/multiple database folders.
shift selection change
The logic for shift selections of thumbnails is different. It now won't deselect anything, and the 'start' position of the shift 'line' is the last focused media. A user suggested this, and it made a lot of sense to me.
full list
- improved my build workflow in several ways, which should result in fewer orphan files and other build weirdnesses
- some bad path usage in initialisation icon resizing has been moved to better temp paths
- hitting page up or down on a manage tags dialog launched from the thumbnail grid no longer clears the current media
- improved file permissions code across the program
- fixed import folders daemon's test code for non-windows
- fixed up some temporary file code that wasn't cleaning up those files when the application was about to close
- fixed a newline parsing problem in copy/paste tags in the manage tags dialog
- added tag cleaning to pasting in the manage tags dialog
- added newline removal into standard tag cleaning process
- fixed a server db bug that was stopping some accounts from being created
- fixed some network session exception creation and catching
- new popup messages should no longer steal focus in most circumstances
- client should recover from serious popup message manager errors better
- hover windows will now only pop up if their media viewer is the currently focused frame
- hover windows will not hide until the mouse moves off them when flash or webm are underneath them
- os x will no longer vanish media in the media viewer on an action like archive or inbox
- fixed juddery media mouse dragging in linux
- improved the way listbooks work to avoid a problem with clientdata in wx linux
- export folder is gone as the default export location–now it is 'hydrus_export' under the current user's home dir
- updated windows ffmpeg to latest version
- fixed an important opencv dll conflict that was causing some gifs to render wrong in windows
- shift focus media logic improved–shift initial thumbnail is now last image selected
- shift selection will no longer deselect anything
next week
With this release, I've cleared out several bugs I had been putting off, so I think I have time to do something fun. I think I'll try to do some custom ratings controls, and if I have time, roll them into a hover window. I've also had an idea for a recycle bin and undo-able file deletion.
Again, unless something special happens next week, I won't be able to work for most of it, which means the next version will most probably come out on the 20th.
Release Tomorrow!
I had an excellent week. I cleared out a lot of important bugs, from unreliable media-dragging in Linux to magically disappearing images in OS X to animation palette noise in Windows. There's a couple of important server fixes as well.
The release will be as normal tomorrow.
Hello there. I have run into an issue scraping from devaintart I get a "typeError" and a failed download. I coulden't find anything in the help or FAQ so i Thought i'd post here.
TypeError
'NoneType' object has no attribute 'getitem'
Traceback (most recent call last):
File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.ClientDownloading", line 1070, in call
File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.ClientDownloading", line 1153, in _GetArgs
File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.ClientDownloading", line 174, in GetFileAndTags
File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.ClientDownloading", line 454, in GetFile
File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.ClientDownloading", line 449, in _GetFileURL
File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.ClientDownloading", line 439, in _ParseImagePage
TypeError: 'NoneType' object has no attribute 'getitem'
Version 155
win32
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v155/Hydrus.Network.155.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v155.tar.gz
I had a good week, although due to fixing some development problems and some IRL stuff, I only had about half the normal time to actually code. I have fixed an array of bugs, and I think my ability to make Linux releases is back to normal.
linux
So, I reinstalled Linux to my laptop and got all the libraries and build scripts working again. I am not sure if today's release is completely good, so if you regularly run the Linux executable, please let me know if you run into problems. I have noticed that the hover window code is buggy, and possibly also the options dialog. I will work on these next week.
full list
- fixed a frame seek error when looping long and/or large gifs with unusual palettes
- improved recovery when fps reads as 0 on videos
- added error detection and graceful recovery and reporting for missing ActiveX flash control
- removed a lot of old imported messaging code that was slowing down boot
- removed some other old library imports that are no longer needed
- gracefully silenced 'application shutdown' daemon errors (they were previously spamming to console)
- fixed a hover window display check that was sometimes spamming linux with edge-case errors
- slideshows will now pause while the right-click menu is open, stopping bizarre full program crashes occuring on many menu items while slideshow was in progress
- fixed a thread_id-sqlite-cache_initialisation problem that was breaking tag archive sync
- harmonised copy/paste tag protocol in manage tags dialog with 'copy all tags' of the tags list
- removed some old messaging code in client db creation
next week
I would love to do some ratings stuff, as I hoped to last week. I have seven things in my 'emergency bugs' to-do, though, so that might cut into my time. I'm hoping just for a normal week, as the subsequent week might have no release due to IRL commitments.
Release Tomorrow!
I had a cluttered but good week. My plans for a ratings hover window were dropped due to lack of time, but I got my OS X and Linux dev environments working again, and I fixed a bunch of bugs. There will be a small but otherwise normal release tomorrow.
clarity
A number of things about hydrus are a bit unintuitive, and the documentation is not helping. I think the basic local functions are also presented in a way that artificially raises the learning curve, but I'm used to it now and can't really remember what I was having problems with. I'll tell you one thing though- having a bunch of tabs hidden behind the "pick new page" under View, when probably most of those items should really be under Download (which is sparsely populated and begging for more items), is an organization nightmare.I set up a hydrus server on my desktop, and configured my laptop as a client. The client can only see things it uploaded- a picture uploaded from the desktop isn't seen on my laptop, pictures uploaded from my laptop aren't seen on the desktop client. Apparently it's going to sync in 28 hours……………… Did my right click > upload, then pending > upload actions not upload the file? If not, what was the popup that told me "upload finished"? If so, why does the client need to wait 28 hours to see a change in the server? Why is there no "sync now" button?! If the file is there why must I wait 28 hours to have it show up in the client?!
The client db and server db are separate, even on the same computer. This means that on the PC running the server, I have to import pictures to the hydrus client, upload them to the local hydrus server, delete them from the hydrus client lest the library be twice as big as it needs to…. then if I ever want to view a picture on the computer where the pictures are in fact stored, I need to "download", eg roundabout copy from hydrus\db\server_files to hydrus\db\client_files? To add insult to injury, trying to view a picture with the internal viewer causes it to "download" but not open- you need to retry opening it after the download. Am I doing something wrong here?
There are two different pages named "manage services" that contain completely different settings. I now understand that one of them is for server.exe, but you would do well to rename one of them to something less conflicting.
Tags and the tag server- I have absolutely no idea what is going on here. If I upload a picture to a server, does that server inherit the picture's tags? I don't think it does (I'll find out in 28 hours, probably), but this means I have to copy all tags from local and paste into the tag server, for every picture. I'm willing to bet this is what "perform a service wide operation" is for, but nothing in the documentation fully explains what "mappings" are and what "copy all mappings to [local] tags" will do. F3 seems to always bring up the local tag editor, too, so I can't easily tag things on the remote end- I have to switch over to remote tags as opposed to just F3 > type tag > enter > move on with life. Or tag locally and periodically export the local tag database to the server.
Tag parents, once created, apparently cannot be deleted. I accidentally added one I did not mean to, and although it is not affecting my library, it's sitting in the parent tag management box with an X next to it. Why won't it go away?
I hope this doesn't come off as hostile, I'm just feeling slightly exasperated and overwhelmed.
Version 154a
win32
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v154a/Hydrus.Network.154a.-.OS.X.-.App.dmg
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v154a.tar.gz
Here's a hotfix for people having update or options-related problems with v154. I've also fixed my OS X dev environment, so here's a release for that as well. My Linux is completely broke, so I'll be reinstalling it tomorrow, and if that all works out, I'll do another short post like this with just the Linux link.
EDIT: Linux is taking a while to build wx, and I was delayed by IRL stuff today. I probably can’t fit it in tonight, and Tuesday would be my next chance to do it, so I’ll probably put off making an ‘early’ Linux release and resync on the v155 release on Wednesday.
By the way, we just hit ten million tags on my public tag repository!
full list
- fixed a bug in v154 update code when there was more than one set of shortcuts to convert
- fixed a faulty default value for num_pixels system predicate that was stopping options from opening
- fixed an error when video fps is 0
Version 154
win32
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v154.tar.gz
Overall, I had a really good week and a half. I am glad I took a couple days off after v153, and I have done a lot since then.
Unfortunately, when putting the release together today, I thought I would update the python install on my Linux and OS X laptops. I did this on Windows earlier in the week, and it took probably sixty seconds, with most of that being finding the link on python.org. Since non-windows is a little more complicated, I figured a half hour tops while things compiled.
Two hours later I was reading through three-year-old bug reports and desperately making symlinks to hack my graphics libraries back together. Everything got fucked up on both computers about five different ways as I tried to reinstall and update other packages for the new version of python. I'm not an expert on either system, so I was really just digging a deeper and deeper hole. I decided to stop banging my head against a wall–I will make a clean install of my dev environments on when I have some time and energy and no deadline, and I will try to remember not to touch anything on Linux or OS X in future unless I have a lot of time to undo it again.
This means that I can't put a Linux or OS X release out right now. If I can fix everything up on Saturday, I will make an auxiliary release then. Sorry!
hover window buttons
The hover window at the top, which I'm not sure whether to call the 'header' or 'command' hover window, now has little icon buttons instead of ugly-as-hell text buttons. They all have tooltips as well. Let me know if anything doesn't work, or if there are any other commands/buttons you would like added.
I've fixed up some sizing and hover-logic issues as well, so animation controls won't be obscured and the hover windows won't overlap.
subs with no file limit
If you tried to make a subscription without a file limit, it may have broken with zero or one file downloaded. I think I found the problem, so if you had this, please go services->manage subscriptions and reset the url cache on the problem subs. Let me know if they are still broken.
default system predicates
The default values for the system predicates will be reset this week. You can change them back to what you like in the options dialog, which looks just slightly different because I updated it to some new clean code I added in the last weeks.
I also fixed a problem with the options panel and system:num_pixels this week, which means system:dimensions is fixed for those for whom it broke.
yaml -> json conversion success
I want my program shortcut support to cover as many button presses and mouse clicks and actions as possible, so I spent some time this week rewriting the way I store and edit lists of shortcuts so they can better support this in future. For now, this ultimately means the custom filter shortcut dialog, which pops up whenever you start a custom filter, works a little less crazily. I have re-written the entire behind-the scenes and replaced the confusing 'save' and 'save as' buttons with a simple 'add/remove' system that'll save any and all changes on OK. The custom filter will start with whatever shortcut profile you have selected when you hit OK.
I want to do more with shortcuts and the custom filter dialog, including harmonising/expanding it to cover all shortcuts for all actions program-wide. I'm thinking something like the shortcut menus you sometimes get in vidya, with tabbed shortcuts for different classes of actions. You probably want to keep your 'navigating' shortcuts the same, but you might want to swap out different sets of shortcuts for assigning ratings or tags, so if I can split shortcuts into pages or tabs, it would make everything simpler. Anyway, if you have any thoughts on this stuff, let me know!
Since I was completely rewriting shortcut storage, which had been stored in the db as YAML, I decided to put some more work into the YAML->JSON conversion I have been considering the past month and make shortcuts the first JSON object I support. After a lot of thought and trial-and-error, I got a really neat system going. For shortcuts, what before took about 30ms now takes 2ms. Even though the raw difference in speed is fractional, the immense speedup of the serialisation process bodes well for when I transfer some of the larger objects (like tag updates) over to my new system.
Now the difficult part of all this is done, I hope to regularly move other currently YAML-able objects over until all the YAML is gone. Things will work faster, produce fewer bugs, take less space, and be easier to update.
db profiling mode
I have added a db profiling mode, under help->debug, that, while active, will measure exactly how much time every database action is taking, and on what lines of code the most CPU time is being spent, and then dump all that information to the log. This is something I do informally while coding a lot, and now you can produce these reports as well. This means that if a particular search is running slow for you, or if a dialog takes a long time to open, you can:
Activate db profiling mode
Attempt the slow database action
Watch your popup windows fill with profiling notifications
Close the program
Check client.log, under install_dir/logs
Scroll to the bottom, where there will be several new profiles, which are basically long tables of numbers
Copy the appropriate table
Email it to me/post it on the board
And then I can get to work on what is being so slow. This is obviously technical, so please do ask me for help if you can't figure out where your log file is, or what db action name you are looking for, or whatever.
full list
- managed to completely break my linux and os x dev environments trying to update python–their release will come soon, once I've cleaned them up
- removed some old cv code
- updated to opencv 2.4.11 on windows
- fixed a bug in trying to upload a small number of tag petitions
- hover command buttons are now shrunk to exact fit size
- added zoom buttons to command button hover window
- added navigation buttons to command button hover window
- added fullscreen switch button to command button hover window
- added archive/inbox button to command button hover window
- added delete button to command button hover window
- added generic close window button to command button hover window
- added tooltips to all the command buttons
- cleaned some misc canvas code
- hover windows now fit better, without overlap
- moved media info strings to top of media canvas, so they hide behind hover header
- hover windows will now not cover webms or gifs when the mouse is over the media container
- hover command buttons now work for inbox filter, including back/skip buttons
- inbox/archive button feels and works correctly for the inbox filter
- updated manage options dialog's 'default file system predicates' mess to use the new cleaner predicate panels
- all file system predicate defaults have been reset to default
- added system:num_pixels to file system predicates default panel
- fixed the system:num_pixels predicate being broken on options save
- removed the ratings system predicates from file system predicates default panel
- fixed misc system predicate bugs
- moved yaml->json conversion forward:
- settled on object code
- added built-in compression to serialisation
- added db tables and access code to support it
- fixed a little layout mess in the edit custom filter action dialog
- created a rich 'shortcuts' class for storing shortcut->action information that can be easily expanded to handle mouse events as well
- new shortcuts class works on new json storage rather than yaml
- moved custom filter shortcut action storage and general handling to new shortcuts object
- moved old favourite_shortcuts tests to new code
- cleaned a lot of the custom filter dialog code
- rewrote the custom filter setup dialog to work more like other dialogs–now all changes are saved on ok, and save/save as/delete is replaced with add/delete
- auto-creation of 'previous' shortcuts is removed, as all changes are saved to shortcut sets anyway
- removed the ratings filters–the numerical filter may make a return in another form, your thoughts would be appreciated
- fixed a cache counting bug when archiving redundant files during import
- sped up file deletion a little bit
- added an inbox cache to reduce laggy inbox checking, particularly search result fetching on dbs with large inboxes
- fixed subscriptions that have no initial file limit, please reset your sub url caches to fix in these cases
- did a lot of server/client database merging and refactoring
- improved some database error handling
- fixed an error in file repository superbans
- optimised some critical db code
- sped up tag censorship filtering
- added a db profiling mode, accessed from the help menu, that will dump copious db profiling info to the log
- misc code cleaning
- improved some popup mesage print sync timing
- rewrote some server auto-setup code to deal with slowly-starting server
next week
After getting a Linux and OS X release out, I want to spend some time on ratings. A ratings hover window in the top-right of the media viewer, with new custom controls for setting ratings with one click.
I also want to copy the new hover button 'toolbar' to the manage tags dialog so you can navigate and interact in the same easy way while keeping the tagging dialog open. This may be more complicated than I expect, so I may have to put it off.
I enjoyed having the extra time for v154, and I was so pleased with the amount I got done that I considered moving to a two-week update process permanently, but the hectic mess of testing this week-and-a-half's huge changelog today and dealing with the several other problems and one annoying db update bug that took five successive builds to nail down reminded me that a week is a good balance between work done and ease of release. It also means a quicker turnaround on bug reports. I'm already several hours later than I am usually, and I think there might be some bad bugs in this week's release just because I was in a rush. If you encounter any problems this week, please let me know and I'll try to have them fixed by next Wednesday!
Speeding up hydrus launch
Pic related it's what I'm practising.
I don't know if this interests anyone else but I've looked a little into speeding up Hydrus' launch:
1. Uninstall pyopenssl if you have it installed. This will shave just over a second off the startup time (if your computer is comparable to mine). Pyopenssl apparently decides to go and parse a bunch of C files when it's imported, which wastes time. I don't know of a better solution than just uninstalling it. https://github.com/pyca/pyopenssl/issues/137
2. If you compile everything in cython (except ClientController and ClientGUICommon, which crash in cython) it will load about 0.5s faster. I can't use very scientific timings here unfortunately, but that's what I've found. The drawback is that you'll need to edit a lot of the code before it works in cython and it will take ages to compile. Seriously that shit takes forever. Probably not worth the half second it gives.
3. Removing unused imports and moving some to a 'lazy' position. The server python files are imported from the client files, but not actually used (for the most part). Moving these out of the way did nothing noticeable to startup time at all. I was a bit optimistic here it seems.
I can't get the full launch/close cycle beneath 5 seconds by any trivial means. cProfile is a bit too rudimentary to give me any idea of what's going on, and RunSnakeRun doesn't run. If anyone has any pointers here to satisfy this one man's startup time fetish I'd be grateful.
BTW, if any Arch users here are following the AUR package, in the next couple of days I might be putting an option in the PKGBUILD for compiling in Cython. Use at your own risk, caveat emptor, etc etc. I will also be switching from using pyc to pyo by default because why the heck not.
Release Tomorrow!
I had a great week and a half. With a lot of time, I got a lot done. I have improved the media viewer's new hovering windows, cleaned up a lot of old code, including several bug fixes and optimisations in important areas, and I have prototyped the first instance of my planned YAML->JSON serialisation protocol conversion in a complete rewrite of the custom filter shortcut dialog and its underlying data.
There is a lot to cover, including a new database profiling mode I want to talk about, so the release post will be a bit longer than usual. Other than that, it should come out as normal tomorrow.
spoilers
Had an Idea I would REALLY like to see, spoilersThere should be an option to set spoiler covers on certain tags, so any image with that tag, when browsing your hydrus catalog, you will only see a spoiler tag. There should be options to make different spoiler lists, for different tags, potentially requiring multiple tags for certain spoilers to pop up, and make them look different, so perhaps you have a list for "spoilers" that pops up on any images with the spoiler tag, and another for "nsfw" that pops up over anything with the nsfw or lewd tags probably needs a way to display multiple spoiler lists
I don't always want to look at porn when browsing my galleries lol
This feature would be an enormous help, I hope its not too hard to impliment
Version 153
win32zip: https://github.com/hydrusnetwork/hydrus/releases/download/v153/Hydrus.Network.153.-.Win32.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v153/Hydrus.Network.153.-.Win32.-.Installer.exe
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v153/Hydrus.Network.153.-.OS.X.-.App.dmg
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v153/Hydrus.Network.153.-.OS.X.-.Extract.only.tar.gz
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v153/Hydrus.Network.153.-.Linux.-.Executable.tar.gz
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v153.tar.gz
I had a good week, finding a lot of time to work on hydrus. I fixed a lot of small things and I added some new things.
system predicates
To save some space, the system predicates for width, height, and image ratio are now combined under a super-predicate system:dimensions. Just hit it and you'll see them lined up in the same dialog. I've also added a megapixel predicate to dimensions, system:num_pixels.
I've also fixed a bug in system:num_tags that was throwing errors when tag censorship included namespaces. num_tags will sometimes be inaccurate for these users until I replace the code that was trying to exclude the censored tags from the tag count.
I've done a lot of behind-the-scenes stuff with system predicates this week, so please let me know if anything draws incorrectly or causes inaccurate results. My tests all ran fine, but I may have missed something.
new hover window
I have added a new hover window at the top of the media viewer. This is an ugly prototype, and I would like to know people's thoughts on what to do with it and how to improve it.
At the moment, common functions are just buttons–I think I will replace them with toolbar-like icons. I also don't like displaying the index string (the 12/120 bit) up top, so I'll probably make another hover window in the bottom right-hand corner, and synchronise it to show at the same time as this new top window.
I'll make the archive button cleverer, as well, so it switches to 'inbox' as appropriate.
I think I'll center and move the 'info' strings up top, so the hover window appears just to move that content to the front, like how the tags seem to.
The tags hover window has some fixes, including better sizing and display code, and it'll update as tags get added or removed, as well.
In doing these hover windows, I think I have broken ratings filters, which are a rarely used part of the program that I want to revisit anyway. I would like to make a ratings hover window next week, in the top-right corner, that'll eventually allow one-click ratings.
I am thinking I may eventually remove the ratings filters. They don't work as intuitively as I originally wanted, and I think a one-click rating would work a lot better. Any filter stuff can be folded into an improved custom filter, I think. If you use them and like them, please say so, and let me know how I can improve them.
new menus
I have added a 'search' menu and added many new quick new page links to the download menu. The link for the F9 page picker is still under view, but anything specific is now under the appropriate menu.
full list
- cut out some out of date stuff from help
- put a nice big red warning at the top of the 'running your own server' page
- added system:num_pixels for megapixel searching
- system predicates height, width, ratio, and the new num_pixels are now collapsed to a super-predicate, system:dimensions
- system:rating will now only show if you have some ratings services
- the system:rating dialog will now hide like or numerical ratings if you have no services of that kind
- complete refactorisation of system predicate dialog
- some misc system predicate bugs and bad panel event precedence fixed
- shortened all predicate spinctrls
- improved how system predicates are stored and used for a search
- moved some view menu items to the download menu and the new search menu
- all of the download gallery pages are added to the download menu
- added some explanatory text to the pixiv dialog
- the tag uploading process will more carefully prepare its subupdates so that mass allocations of the same tag to many files will be split into smaller chunks, avoiding connection timeouts in these cases
- hover windows will no longer hover over flash files
- tag hover frame will resize a little based on the canvas width
- the twenty pixel mouse warping when the mouse starts dragging close to the edge of the media viewer will no longer occur on flash files
- the canvas buffer for flash files is expanded to five pixels either side
- the hover tags window will now update itself when the media's tags change
- new hover window for details and commands, up top of the media viewer
- moved the old popout windows' buttons to the new hover window
- added hover window commands for the normal browser
- cleaned up some general canvas code
- index string will now display with a slash in all cases (it was a backslash for windows, for accidental reasons)
- file limit for subscriptions and download pages now has minimum value 1
- searching for number of tags while tag censorship has some namespaces will no longer throw an error
- moved boot and exit code responsibility from the splash frame to the controller
- improved the boot and exit code generally, including dragging feedback lag and error handling
- improved some thread error handling
- fattening service info won't trigger on client shutdown quite so much, which should speed up shutdown a little
next week
I have a lot to do. More hover window stuff and more bug fixes, most likely. I didn't find time for any YAML->JSON stuff this week, as it seems to be a bigger job than I can do in small bits. I might just have to schedule a whole future week and do it all in one go. I want to clean and merge my client and server db code, so that'll probably be my saturday.
Then again, I think I burned myself out the past two weeks, so I might take a bit of time off.
No Release Tomorrow!
I took some time off this week to destress and fix some IRL problems. As a result, I only had today to work on hydrus. I've converted the new header hover window to use icons instead of ugly text buttons, but that's it. Rather than spending time putting a small release together tomorrow, I would rather just keep working.So, there will be no release tomorrow. I am now aiming for the 22nd.
Version 152
win32zip: https://github.com/hydrusnetwork/hydrus/releases/download/v152/Hydrus.Network.152.-.Win32.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v152/Hydrus.Network.152.-.Win32.-.Installer.exe
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v152/Hydrus.Network.152.-.OS.X.-.App.dmg
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v152/Hydrus.Network.152.-.OS.X.-.Extract.only.tar.gz
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v152/Hydrus.Network.152.-.Linux.-.Executable.tar.gz
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v152.tar.gz
I had a good week overall. I fixed a lot of stuff and added some things.
hovering frames
As I planned last week, I have prototyped a new kind of 'hovering' window in the media viewer. Now, whenever you put your mouse on the left side of the media viewer, a tags list will pop up. It will hide again as soon as you move you mouse out. Now when you have a large image that would normally hide your tags, you can see them just by moving your mouse over. The hovering frame is interactable, so you can right click for a normal tag menu, like in other places.
This is a very basic proof of concept. I quite like the feel of it, and I would like to do more, perhaps adding a toolbar, at the top. I might also make left-clicking on the new hovering tags launch the manage tags dialog. And I'll probably fine-tune a lot of the logic behind when to show and hide these things. Doing quick ratings is high on my list as well. The sky is the limit, really. If you have your own ideas about what might be useful here, or different or custom behaviours for any of it, please let me know! This stuff will likely lead to some significant changes in overall workflow, hopefully much for the better.
download file limit
Subscriptions and regular gallery download pages now have a custom file limit, defaulting to 500 but optionally limitless, that will tell the downloader to stop fetching new pages of results once that value is hit. This should allow fire-and-forget searches for big queries.
For subscriptions, the limit only applies to the initial synchronisation; after that initial run, it will download however many new files have appeared to stay synched. This lets you hit the ground running on a big query and subscribe to 'new files' rather than the hundreds of pages of back catalogue.
In writing this, I just realised that setting this to 0 will possibly cause big problems. Don't set it to 0! I'll set the minimum to 1 next week.
full list
- added prototype hover frame for tags in media view
- hover frame shouldn't show when a dialog is open
- manage tags dialog launched from media viewer now has a delete button
- subscriptions now have an 'initial limit' variable, defaulting to 500, that limits the total number of files the subscription daemon will look for on the initial sync
- added a similar file limit spinctrl to gallery download pages
- updated layout of import files dialog
- cleaned up some ugly global variable scope stuff
- fixed initialisation of advanced import options in file import dialog
- made a good start to better object serialisation
- subscription and repository sync daemon jobs that stop due to a dialog-driven change during processing will now cancel themselves after five seconds
- fixed namespace (e.g. 'series:') tag censorship
- fixed the fullscreen switch bug that was breaking an (initially fullscreen)->(non fullscreen) media viewer
- fixed some search logic (some system predicates were not firing when there were no regular tags present)
- removed some artificial delays on daemon db access, let's see if it chokes anything
- harmonised a bunch of client and server controller code
- created a common controller class and merged a lot of the client and server controller code into it
- general code cleanup
- more general code cleanup
next week
My YAML-to-JSON conversion had a good start this week, so I want to continue that a bit, and I'll also do more of this hover stuff. I have about twenty small cleanup jobs in my immediate to-do, so I'll probably focus on those as well, just to get them out the way.
so i formatted and installed shitty winfag 8.1 with python 2.7.9 and whatnot. trying to run hydrus now gives me this error:
Traceback (most recent call last):
File "<string>", line 20, in <module>
File "C:\code\pyinstaller-dev\PyInstaller\loader\pyi_importers.py", line 270, in load_module
File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.ClientController", line 9, in <module>
File "C:\code\pyinstaller-dev\PyInstaller\loader\pyi_importers.py", line 270, in load_module
File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.HydrusServer", line 2, in <module>
File "C:\code\pyinstaller-dev\PyInstaller\loader\pyi_importers.py", line 270, in load_module
File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.ClientConstants", line 9, in <module>
File "C:\code\pyinstaller-dev\PyInstaller\loader\pyi_importers.py", line 270, in load_module
File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.HydrusFileHandling", line 13, in <module>
File "C:\code\pyinstaller-dev\PyInstaller\loader\pyi_importers.py", line 270, in load_module
File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.HydrusImageHandling", line 4, in <module>
File "C:\code\pyinstaller-dev\PyInstaller\loader\pyi_importers.py", line 270, in load_module
File "C:\code\Hydrus\build\client\out00-PYZ.pyz\cv", line 1, in <module>
File "C:\code\pyinstaller-dev\PyInstaller\loader\pyi_importers.py", line 414, in load_module
ImportError: DLL load failed: The specified module could not be found.
It doesn't even say which dll is missing. How do I fix this? Hydrus currently doesn't work at all.
Release Tomorrow!
I had a good week and got a lot done. I added a num_pixels system predicate, fixed quite a few bugs, and prototyped a new hover window.
How much grunt work have you contributed?
What type of files are you tagging? Much of my collection is sorted by creator so I do a lot of creator: tagging.
Pixiv
How do I download pixiv galleries with Hydrus? I don't see a "Pixiv" button next to all the other choosable sites (HF, Newgrounds, etc).There's "manage pixiv account" under Services, but I don't know where to go from there.
Any help would be appreciated
Version 151
win32zip: https://github.com/hydrusnetwork/hydrus/releases/download/v151/Hydrus.Network.151.-.Win32.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v151/Hydrus.Network.151.-.Win32.-.Installer.exe
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v151/Hydrus.Network.151.-.OS.X.-.App.dmg
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v151/Hydrus.Network.151.-.OS.X.-.Extract.only.tar.gz
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v151/Hydrus.Network.151.-.Linux.-.Executable.tar.gz
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v151.tar.gz
I had a good week. I cleaned and wrote a lot of code. Most of my time went into boring things that are good but don't change the look or feel of the program, but I did also fix a couple of important bugs, and I made tag archives more powerful.
no booru api integration
I wanted to extend my booru parsers to interface with The Booru API™. After some research, I decided not to do it yet because:
1) Although the different boorus that have APIs are obviously trying to follow a few distinct standards, in practise some of them use different calling urls and parameters, and they serve their data in different markup languages and in different layouts. In order to support any decent number, I would have to write a very generalised API that would take a whole load of work.
2) The one thing they all did do was serve tags without namespaces! Most boorus, of course, do not explicitly label their namespaces, and instead colour them on the webpage. I currently harvest this information by looking through the HTML for the CSS class and prepending the contained text with 'series:' or whatever. So, absent some overly complicated scheme like parsing a booru's master taglist and swapping tag aliases like hydrus does siblings, in order to get namespaces from booru API information, I would have to parse the html page the image was on, which is what I do already. This means any booru interface I wrote would only save a bit of time on the initial gallery page parsing, and that's it.
Ultimately, I decided my time was best spent on other things. If the boorus get a bit more standardised, and if they include namespace information in their APIs, I'll revisit this topic.
bugs
If you have been getting full program crashes after navigating the media viewer with the manage tag dialog's new previous/next buttons, I am fairly confident I have fixed it.
Also, I introduced an odd bug a week or two ago when I collapsed some constants together that meant opening the import files dialog would bork a lot of future gui layout and sizing. I had planned to look at this problem next week, but then I encountered it while making this build, and seeing it was so easy to replicate, I made a quick fix. This means the import files dialog looks a bit ugly right now, but it won't break anything else. I will make it look neater next week.
Import folders will now ignore zero-length files (which are sometimes set up by backup/sync software), so if you have these, you shouldn't get any more error spam from them.
tag archives
Under services->review services->tag service->perform a service-wide operation, there is a new button to one-time add or delete an external tag archive's tags, by namespace. The same dialog already lets you export to tag archives, so I hope this makes one-shot mass-tagging a bit easier for everyone!
This is also useful if you want to quickly copy your local tags from one client to another.
You can only delete tag archive tags if the service is 'local tags' or it is a tag repository and you are a janitor (i.e. you have 'resolve petitions' permission). This is because doing a normal user's mass delete would need to create petitions for every tag->files mapping, and the current admin workflow is not set up to quickly deal with 300,000 petitions!
I have added a couple of useful functions to the hydrus tag archive class and also optimised the hydrus client database's code that interacts with them; doing a large operation will now take a few more seconds to calculate, but the actual sync should be a lot faster, and it will also include non-local files, which were not previously being checked.
full list
- added a possible solution to the manage tags dialog next/previous buttons crash
- completely overhauled temporary file management throughout client and server
- removed old temp folder
- copy files to clipboard will now copy the database's exact file paths (rather than copying to a temp folder beforehand), making it a lot quicker
- the import folders daemon will no longer attempt to import (and hence spam errors about) zero-length files
- added one-time tag archive sync button to the 'perform a service-wide operation' dialog in review services. this lets you add or delete the archive's tags by namespace
- revised the tag archive sync code a little so it syncs to non-local files as well, if the hashes can be lined up
- added hash cross-referencing to archive sync code before maina processing step, often saving a huge amount of time
- fixed a parsing error with advanced content updates at the gui-end
- added GetName and IterateHashes to hydrus tag archives
- added default 'similar to' max hamming distance to file system predicate defaults in client options
- fixed up the default text display of the file service system predicate
added some text to the sort/collect options panel to better explain what the big listbox is for
- fixed a popup numerical typo when uploading files to a repository
- fixed an utterly bizarre bug caused by the import files dialog that was screwing up subsequent dialogs' layout and size, apparently by wx voodoo
- refactored all the functions and classes of hydrusconstants to better locations
- misc other refactorisation
- misc cleaning
- renamed hydrusdownloading to clientdownloading
- changed all default { } and [ ] parameters to None. I don't think it mattered, but just in case for the future
- moved some ugly global variable stuff to better places, hoping to do more in future
- cleaned up many unused imports
next week
I really want to rework my media viewer so the background tags and text float above the current media as you move the mouse. I'd also love for interactable elements like the tags dialog to be more fluid as well, so a simple mouse gesture lets you start managing tags. This is likely too complicated for one week's work, but I think I can get the floating done.
Also, I have decided to move from YAML to JSON for my general storage and transmission protocol. I did some 'normal usage scenario' tests this week, and JSON was sometimes 200 times faster. JSON isn't as flexible as YAML, but it means tag update processing will spend all that much less time on the 'parsing update' step. I want to make a plan for moving over, and also for improving how the whole system works, making it easier to do db updates when something I am storing changes between versions.
Release Tomorrow!
I had a busy but good week. I cleaned code, fixed bugs, prototyped a new floating window in the media viewer, and added a neat 'number of files to download' limiter to subscriptions and gallery download pages.
Version 150
win32zip: https://github.com/hydrusnetwork/hydrus/releases/download/v150/Hydrus.Network.150.-.Win32.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v150/Hydrus.Network.150.-.Win32.-.Installer.exe
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v150/Hydrus.Network.150.-.OS.X.-.App.dmg
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v150/Hydrus.Network.150.-.OS.X.-.Extract.only.tar.gz
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v150/Hydrus.Network.150.-.Linux.-.Executable.tar.gz
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v150.tar.gz
I had a very good week, overall. Building on the bit of work I fit in in the week before, I mostly fixed and improved things, and did a lot of code cleanup. Most importantly, I think I have fixed the problem of slow tag update processing.
tag update processing
I have added a 'speed mode' for repository update processing–it will kick in when the client thinks it is idle, which by default means after half an hour of not opening dialogs or looking at images. This speed mode will use a lot more resources than usual, but in turn will make the gui extremely laggy (hence why it only kicks in when you aren't doing anything). It processes tags about four times faster than normal, meaning about 3-4 million mappings per hour. Given that a few months ago, we were at something like 100,000 per hour, I am very pleased with this progress.
On a normal PC, and assuming the user lets the client alone to get on with it, syncing my entire public tag repository should now take about 2.5-3 hours. 4 hours was my aim, so I am content that this problem is no longer a problem. As tag repositories break ten or twenty or fifty million mappings, and so the clients' sync time slows even further, I will work on this problem more, but for now, the biggest bottleneck in update processing is inefficient use of bandwidth and laggy YAML parsing.
serious gallery downloader problem fixed
I have fixed a serious gui-lockup that no one has reported but blew up in my testing today. I'm hoping I only exposed it in this week's work, but if your client ever locks up completely just as a new downloaded image was fading in, that problem is fixed in this version, so make sure to update.
full list
- added an 'idle' updating processing mode that churns through tags four times faster but makes gui laggy. it will kick in whenever you haven't done anything in thirty minutes
- improved some gallery downloader status texts
- gallery downloader will no longer wait 5 secs before fetching first page of urls
- fixed a critical race condition that had a chance of causing complete and unrecoverable gui freeze in the gallery downloader code
- fixed rendering of some bizarro pngs that CV couldn't understand
- fixed tag parent management for admin users
- fixed up some tag parent dialog code
- tag sibling dialog now accepts single reasons for multiple pairs, like the tag parent dialog
- improved error state on rendering video that reports resolution of 0x0
- removed looping error reporting on certain static image rendering errors
- fixed an error when some thumbs were 'collected' while fading in
- tagboxes now select on home/end and support numpad versions of page up/down
- fixed a radio button initialisation bug
- boosted the initial size of the 'set default advanced tag options' dialog so the collapsible pane is more visibile
- silenced a upnp error when external ip can't be parsed
- reduced some tag dialog previous/next button lag
- general code cleanup and refactorisation
- gui flag refactorisation and unification
- refactorisation of most non-constant stuff from constants files
- refactorisations of media mixins class in prep for a general rewrite of that code
- improved client initialisation of default options, boorus, and imageboards
next week
Now update processing time is less of a concern, I am free to choose another big thing to work on. Someone mentioned a booru API, so I think I will look at that, to make booru processing more efficient and polite (atm, booru stuff works by parsing the gallery pages' html, which is not ideal). I also have the usual ton of issues to deal with and code to clean up.
Release Tomorrow!
I had a pretty good week. The booru API upgrade didn't come together, but I did a bunch of general improvement, fixed some important bugs, and added some neat new tag archive functions for easier mass-tagging.
Crash after Manage Tags Next/Previous Button use
This thread is for people experiencing a full program crash after using the next/previous buttons in the manage tags dialog. If you haven't told me you have it yet, please post here with your experience with this bug. What OS are you using? Does the crash happen every time?If you have told me, please monitor and post to this thread as we work for a solution.
Cap and catch up subscription downloads?
Can we get a way to cap subscription downloads and catch up the URL cache so it doesn't try to download older results? I do a lot of high-result searches where I'm only interested in the newest files.
Version 145
win32zip: https://github.com/hydrusnetwork/hydrus/releases/download/v145/Hydrus.Network.145.-.Win32.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v145/Hydrus.Network.145.-.Win32.-.Installer.exe
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v145/Hydrus.Network.145.-.OS.X.-.App.dmg
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v145/Hydrus.Network.145.-.OS.X.-.Extract.only.tar.gz
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v145/Hydrus.Network.145.-.Linux.-.Executable.tar.gz
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v145.tar.gz
This is obviously my first attempt at an 8chan release post. If the formatting is completely screwed up, I might have to edit a couple of times.
I had a pretty good week, overall. I nuked several things off my to-do list and I completed several non-code project tasks, such as setting up this board.
custom gui colours
In file->options->colours, you can now edit some basic gui colours–thumbnail and media background colour, that sort of thing. Please let me know if the colour buttons do not work for your platform.
previous and next buttons on manage tags dialog
The manage tags dialog (if launched from the media viewer, not the thumbnail view) now has <- and -> buttons, which will virtually 'OK' the dialog's current changes and browse the media viewer in the background. Try it yourself, and you will see what I mean. You can also press Page Up or Page Down from the autocomplete entry field to achieve the same thing.
I hope this will make tagging while browsing a bit quicker and easier!
fit to canvas
Under file->options->media, there is now an option to automatically zoom smaller images to the full size of the media viewer canvas as soon as they are loaded. I'm really pleased with the effect of this, even though it is simple.
8chan board
My old forum is a bit creaky and doesn't do anonymity in quite the way I would like. I have been thinking about making an 8chan board as a replacement for a little while, and a user prompted me this week to do it, so I did, and here we are. I've added links to here in the client and my help, and removed those to my old forum.
I have left the old forum up for now, but if this board continues to work out, I will probably take it down in a few weeks.
I will try to answer all posts that ask for my input, but if I miss something for several days, please do email me or repeat your thread or whatever seems appropriate to get my attention.
full list
- added custom gui colours for thumbnail backgrounds and borders, the autocomplete background, and media background and text
- added <- and -> arrows to manage tags dialog launched from navigable media viewer
- on empty input in the manage tags dialog, page up and page down work as shortcuts for the new <- and -> buttons
- fixed the media height calculation for animations, so when they are vertically scaled, the total height including scanbar won't overflow off screen
- allowed non-integer page, chapter, volume tags in display and sort calculations
- semi-integer tags will sort along with integer tags and string tags like so: 0 < 0a < 0b < 1 < 2 < 22
- improved the old tag/media sorting code
- removed loli and shota from hentai foundry filter options
- patched old db-stored predicates to attempt to convert to the new format when queried for _inclusive
- this _should_ have fixed the recent export folder problems
- created an 8chan board, and updated my various links, including in the client, to migrate from my old forum to this
- added a 'fit to canvas' checkbox in file->options->media that will automatically zoom small images to the full size of the media viewer
- misc code improvements
next week
I want to speed update processing up further, including perhaps beginning a 'swift mode' for the initial sync to a repository, or maybe to any process of >10 updates or something.
I would also like to improve some of the behind-the-scenes of subscription processing, so that pausing, cancelling, and reviewing of subscriptions is a generally more efficient and data-rich experience.
Other than that, I just want to process my smaller-things to-do so I feel better about spending some time on a big thing in a couple of weeks.
Version 149
win32zip: https://github.com/hydrusnetwork/hydrus/releases/download/v149/Hydrus.Network.149.-.Win32.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v149/Hydrus.Network.149.-.Win32.-.Installer.exe
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v149/Hydrus.Network.149.-.OS.X.-.App.dmg
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v149/Hydrus.Network.149.-.OS.X.-.Extract.only.tar.gz
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v149/Hydrus.Network.149.-.Linux.-.Executable.tar.gz
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v149.tar.gz
I had a good week. Another raft of bugs are cleared, and a lot of old bad code is replaced with something better. I also made an important optimisation that makes all sorts of things run much faster.
db optimisations
I finally got around to improving update processing this week, so I ran some thorough profiling and discovered that under certain circumstances, a bit of my db code that usually took ten microseconds would sometimes bloat up to seventy milliseconds or more. This bit of code is used all over the place, and was adding lag in places that expected it to be very quick, and so several operations were running far slower than they should. Furthermore, the variability of the slowdown threw off some time-balancing code, adding lag to the gui.
I rewrote the code to be more sensible and reliable, and a couple other small improvements, and voilà, update processing now runs about twice as fast! Other things, like general media fetch after a query, are running anything from a bit faster to ten times as fast. I load thumbnail results at about 4,000 per second, which feels like hyperspeed as compared to before.
As a result, I think a full sync of my public tag repo should now take about ten hours on a normal pc. I'm pleased with that, but I already know some more places where I can save time, particularly when the client is idle. I would love to get it down to four hours, which would be about 2 million mappings per hour. I will continue working on this in the coming weeks.
I have put in a quick temporary fix in this code to skip one of the slowdowns, so please let me know if you see any weird new errors while repositories sync. I will make a better permanent solution in the near future.
full list
- all listctrls with an attendant delete button will now support the same action with the delete key
- several normal tag boxes will support the delete key as well
- added a temporary service cache to the client db to speed up service fetching until I can figure out a yaml replacement
- cleared out inefficient tag and hash database fetching and generation code
- tag parent dialog will now ask for an action/reason for every ( children, parent ) pair rather than every ( child, parent ) pair
- fixed an error with collecting by rating
- fixed the 'fix' to the gui_colours db update problem
- the autocomplete dropdowns in the paths-to-tags dialog will no longer produce errors when tags are submitted
- improved accuracy of taglist scroll-to-selected scrolldown calculation
- collapsed some of the helter-skelter db code in prep for future improvements
- moved daemon code to separate files
- silenced server upnp errors
- all except: continue lines are banished!
- cleaned up some miscellaneous exception code
- cleaned some ugly behind the scenes a/c dropdown button code
- did some general code cleanup
next week
I'm buried in bug reports right now, so I need to spend a week mostly catching up. Beyond that, I have a lot of small things to add or improve. I'm also going to explore a replacement for YAML, my current data serialisation protocol, which is proving just too slow for my purposes.
Version 146
win32zip: https://github.com/hydrusnetwork/hydrus/releases/download/v146/Hydrus.Network.146.-.Win32.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v146/Hydrus.Network.146.-.Win32.-.Installer.exe
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v146/Hydrus.Network.146.-.OS.X.-.App.dmg
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v146/Hydrus.Network.146.-.OS.X.-.Extract.only.tar.gz
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v146/Hydrus.Network.146.-.Linux.-.Executable.tar.gz
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v146.tar.gz
I am busy with some IRL stuff right now, so this week's release is a quick cleanup of all the bug fixes and other small things immediately in front of me before I catch a flight. I will be away from my dev computer all of next week, so I will not be able to do another release until the 18th.
better zoom code
I won't describe every point this week, as they are all fairly self-explanatory, but I'll highlight this, because I am pleased with how it works:
The 'fit to canvas' zoom is now inserted into the list of zooms, so if an image starts at 93% zoom, and you zoom in to 100% and then zoom out again, it'll zoom to 93% (rather than immediately to 90% as before). Furthermore, pressing 'z' will also jump between 100% and this 'fit to canvas' zoom value for all images, including those smaller than the canvas size.
full list
- manage tags and ratings dialogs will now OK on F3/F4, not CANCEL
- zoom switch (default shortcut 'z') will now work for images smaller than the media viewer's canvas
- in the media viewer, the canvas-fitting zoom value is now inserted into a media's scrollable zooms
- added 'tags box background' gui custom colour
- volume, chapter, and page tags will sort properly again in thumbnail view, including last week's improvements
- the thread watcher will no longer break its checking loop on non-404 http request or parse errors (i.e. the manual 'check now' button will still work after a misc error)
- to remove clutter, the preview window will no longer show tags and file information like the normal media viewer
- subscriptions will now show a live 'show files' button as they process
- the copy and 'open new page' right-click menu options on tags will now work on more controls
- cleaned quite a bit of tags box code
- cleaned up and unified a lot of miscellaneous canvas zoom and display code
- fixed an error when trying to upload a file petition without any accompanying file uploads
next week
Again, there will be no release on the 11th. I'll have some internet next week, so I should be able to reply to emails and posts; I just might be delayed a little. For the release on the 18th, I would like to do what I originally planned after v145, which is improvements to update processing and subscription/downloader code. My IRL stuff should be completely done at that point, so I feel confident and enthusiastic about a return to a normal work schedule.
We hit 7 million mappings on my public tag repository this week. Thank you for contributing!
Also, a user made a tag archive of derpibooru, if you are interested. He did a great job of parsing the tags and merging all the namespaces!
http://www.mediafire.com/download/tyfbh60fv3cfndf/derpibooru.7z
Regex thread
Post your regexes for tag importing here.This regex is a better filename regex from the Perl Cookbook. The current regex in Hydrus won't catch brackets, hyphens or parens, this one will catch any character that's valid in a Windows filename.
[^\\/:*?"<>|\r\n]+(?=\..*$)
Release Tomorrow!
Overall, I had a really good week. I did a heap of code cleanup, fixed some bugs, improved how the tag siblings dialog works, and I sped up tag processing by about four times (about 3.5ish million mappings an hour, now!) when the client is idle.The release will be as usual tomorrow. I will have to do a bit more testing than usual, but absent any big problems, it shouldn't delay me.
Undo tag archive add
So, with all of this tag archive stuff we've been doing, it occurred to me that it might be good to have a way to comprehensively un-add a tag archive. That is to say, "remove any tags from my Hydrus library which match this DB".This way if someone uses an old version of a database and wants to replace it with a new one, they can do so without detritus hanging around.
I think that currently if you remove a DB you keep the tags - right? So maybe this could be an option upon removal.
No Release This Week!
I had an unexpectedly busy week, which meant I could not spend much time working on Hydrus. I've fixed a handful of bugs, but it isn't enough to justify making a release.I will now aim for March 18th.
I had a pretty good week. I cleaned a lot of code and fixed a lot of bugs, and I made some very important optimisations to a few database queries that has massively sped up update processing and media fetching (the time between you putting in a query and the thumbs fading in on screen).
On my dev machine:
Updates now process at 800k tags per hour (about twice as fast).
Media fetching now works at about 4,000 files per second (probably about four-six times as fast, I think).