[ / / / / / / / / / / / / / ] [ dir / agatha2 / caco / choroy / dempart / jenny / tingles / vril / wmafsex ]

/hydrus/ - Hydrus Network

Bug reports, feature requests, and other discussion for the hydrus network.

Catalog   Archive

Winner of the 75nd Attention-Hungry Games
/caco/ - Azarath Metrion Zinthos

March 2019 - 8chan Transparency Report
Comment *
Password (Randomized for file and post deletion; you may also set your own.)
* = required field[▶ Show post options & limits]
Confused? See the FAQ.
(replaces files and can be used instead)

Allowed file types:jpg, jpeg, gif, png, webm, mp4, swf, pdf
Max filesize is 16 MB.
Max image dimensions are 15000 x 15000.
You may upload 5 per post.

New user? Start here ---> http://hydrusnetwork.github.io/hydrus/

Experienced user with a bit of cash who wants to help out? ---> Patreon

Current to-do list has: 1,626 items

The program is now on Python 3! Check v335 release post if you need to update from before then!

Current big job: None--check the pinned poll thread

File: 3711d0607c5673e⋯.mp4 (3.38 MB, 1024x576, 16:9, 3711d0607c5673ea2d3f398a9a….mp4)

ff909c  No.12152[Reply]

Ok lads, as I am now finishing up OR search, I am soon going to be free to work on a new 'big job'. I am pleased that I was able to make simple Client API and OR search in much faster iterations than previously. I hope to continue like this, keeping the next big job 8-12 weeks at the most before running a new poll.

The current list is:

Just catch up on small work for a couple of months

Reduce crashes and ui jitter and hanging by improving ui-db async code

Clean up code and add unit tests

Improve tag siblings/parents and tag 'censorship'

Add ways to display files in ways other than thumbnails (like 'details' view in file explorers)

Add text and html support

Add Ugoira support (including optional mp4/webm conversion)

Add CBZ/CBR support (including framework for multi-page format)

Add import any file support (giving it 'unknown' mime but preserving file extension)

Improve 'known urls' searching and management

Explore a prototype for neural net auto-tagging

Add support for playing audio for audio and video files

Add ui for waifu2x and other file converters/processors

Write some ui to allow selecting thumbnails with a dragged bounding box

Add popular/favourite tag cloud controls for better 'browsing' search

Improve the client's local booru (this likely now means a backend migration to the Client API)

Improve duplicate db storage and filter workflow (need this first before alternate files support)

Improve shortcut customisation, including mouse shortcuts

Add ratings import/export, and add 'rating import options' to auto-rate imports

Add more commands to the undo system

Improve display of very large/zoomed files in the media viewer

Set thumbnail border colours on user-editable rating and namespace conditions

Post too long. Click here to view the full text.
102 posts and 9 image replies omitted. Click reply to view.
Post last edited at

6d6dd6  No.12348

Are there any plans at all to (at least have the option to) keep metadata of individual tag mappings, like a file's tag history or keeping track of which process added this tag to this file?

Like was it typed manually, was it imported while scraping, etc.

I assume this would bloat the DB by a big factor but at least for my personal use I think that might be worth it down the line

YouTube embed. Click thumbnail to play.

71afbb  No.12289[Reply]


zip: https://github.com/hydrusnetwork/hydrus/releases/download/v348/Hydrus.Network.348.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v348/Hydrus.Network.348.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v348/Hydrus.Network.348.-.OS.X.-.App.dmg


tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v348/Hydrus.Network.348.-.Linux.-.Executable.tar.gz


tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v348.tar.gz

I had a good week. It is mostly small updates and fixes.

all misc this week

After the recent weeks' thumbnail work, I became frustrated with how laggy the client's file system could get under heavy load. This week I have written a new file access locking system that has less latency under heavy simultaneous use and is also safer for certain edge cases. Multiple import queues and thumbnail fading and regular media browsing will all interact with less lag now.

The Client API has several improvements: the /add_tags/add_tags call has its 'hashes' parameter fixed and now does sibling-collapse and parent-expansion on its 'add' tags (which caPost too long. Click here to view the full text.

18 posts and 3 image replies omitted. Click reply to view.

63708f  No.12341

File: ecff8ed9b1a042b⋯.jpg (31.28 KB, 619x239, 619:239, Capture.JPG)


here I changed the namespace from "creator" to artist. When I enter the URL, Hydrus no longer fetches the artist at all.

File: 1426721772716.png (100.78 KB, 1624x1081, 1624:1081, 1327614072601.png)

7f2c0e  No.471[Reply]

Drag and drop windows with tag rules. Show two windows side by side and one window can be programmed with the rule "ADD tag foo" and the other one has the rule "REMOVE tag foo, ADD tag bar" and you can drag and drop files to them.

Deriving tags from regex of other tags/namespace tags. A file has the tag "filename:big_ugly_name" and we could regex that namespace for another tag.

Tag sets with hotkeys: save a set of tags under a hotkey so it's quick to add them to a file while filtering

Opaque window behind tag list in the corner so it doesn't get hidden by picture background

Option to default certain mime types to be excluded from slideshow and only open externally, will help with videos with odd codecs that don't preview in the slideshow correctly

Option to specify hamming distance in "find similar images", you can't change the option once it's in the filter window and you have to enter the hash manually in the "system:similar to" option
657 posts and 192 image replies omitted. Click reply to view.

5d012e  No.12099


Actually, I forgot the hydrus downloaders also grab url's which seems like a better option for exporting images. Would be cool if you could export based on url's, maybe some options to trim them down so you can get the original file name.

e339aa  No.12100

So lately I've been manually tagging a number of related images. The fucking clear button has become the bane of my existence.

Want to add 5 or 6 of the most recently added tags? You accidentally clicked the clear button because it is literally one line above the most recently added tag and have to manually type them all out again.

Can we either move the clear button to the bottom of the window so you can't accidentally click something you only need on rare occasions, or add a confirmation dialog?

690b74  No.12151

Working suggestion: make GUI font a little more hackable. To do that, add to the ClientConstants.py:

def get_default_font():
return wx.SystemSettings.GetFont(

And then run magic

sed -r -i 's/wx\.SystemSettings\.GetFont\( *wx\.SYS_DEFAULT_GUI_FONT *\)/CC.get_default_font()/g' *.py

and commit the changes.

This will make it easier to, both, add a setting for the fonts, and make it easier to patch the font in the code.

7d0642  No.12170

It would be useful for mp3 files to use their embedded album cover art as a thumbnail if one exists.

d599a3  No.12349

Minor UX tweaks to prevent accidental duplicate marking:

Ignore mouse clicks when another window is in focus, just switch focus to hydrus instead of confirming a dupe.

Ignore mouse clicks for the first couple of milliseconds of a new comparison, to prevent double clicks from being treated as marking an unseen image as a duplicate

File: 5e7a84ad6d37996⋯.webm (6.59 MB, 320x240, 4:3, 5e7a84ad6d37996be590d298e….webm)

4e1412  No.12347[Reply]


File: a8e4b36af39226c⋯.jpg (47.38 KB, 475x351, 475:351, shovelingcoal.jpg)

ffaaf9  No.12346[Reply]

get mumble cucks

c11c93  No.12345[Reply]

hydrus is shit nigga just use folders and subfolders

my 12345 says i'm right

File: 3f8a8d22364b1de⋯.jpg (111.13 KB, 400x400, 1:1, 1394958218797.jpg)

d5679d  No.9327[Reply]

Here you can ask questions so that the board is not clogged with small threads.

>>6021 has reached its bump limit, so I made a new thread.

687 posts and 126 image replies omitted. Click reply to view.

4cd362  No.12311


Main problem was setting up a workflow that compared 'appropriate' files against each other. I forget the exact old implementation, but I think it just presented random pairs. Asking whether a picture of a cute elf is 'better' than a screenshot of a thread proved completely unhelpful. Also to 'accurately' rate a file on a ten-star scale, it was like 4-6 decisions, which on 100,000 files means way too much work, especially when the comparisons were so bizarre.

I think if I were ever to bring a system like that back, I would have to use a search or something, like how I am going to add to dupe filter this week, so you'd say 'ok, let's rate some elf pictures against each other'.

The whole idea is on the back-burner for now. Just clicking your estimate of whether something is 4- or 5-star is quicker and gives more control for subjective adjustment than the last attempt at 'objective' rating did. I could bring it back, but I am still waiting for a better solution. It is still vapourware, but maybe in five years we'll be able to teach a neural network to recognise ratings, or more likely rating estimates (like 'I know you like tag x, y, z, so I'd estimate this file that has those also is prob 4-5 stars), to massively reduce the workload here.

6b615c  No.12338

If you're using the downloader to grab an image that you already have (for the purpose of tags), is it grabbing the image again or just the tags? Just wondering for the sake of bandwidth

cd1464  No.12342


what's 9+10

cd1464  No.12343

why is your software so bloated?

cd1464  No.12344

what language is this program made in

YouTube embed. Click thumbnail to play.

1cdbfc  No.12153[Reply]


zip: https://github.com/hydrusnetwork/hydrus/releases/download/v347/Hydrus.Network.347.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v347/Hydrus.Network.347.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v347/Hydrus.Network.347.-.OS.X.-.App.dmg


tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v347/Hydrus.Network.347.-.Linux.-.Executable.tar.gz


tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v347.tar.gz

I had a good week. OR search is essentially finished, and I cleaned up and fixed a variety of other things.

A new 'big thing to work on next' poll will be going up soon. If you are interested, please check out the discussion thread here:


or search

As previously discussed, I have moved OR predicate construction to the standard dropdown list below the tag input. It now appears as the top result, where you can hit enter on it to submit it as-is. Also, while under construction, a 'cPost too long. Click here to view the full text.

18 posts and 3 image replies omitted. Click reply to view.

1cdbfc  No.12208


Oh yeah, your odds of getting the same image in a row are fairly high, as my current 'get some dupes to process' job batches decisions together into various similar types. You'll likely get several pairs in a row that feature the same file as A or B.

1cdbfc  No.12209


Yeah, it'll say the action as well. In the filter it will be something like:

"Deleted in Duplicate Filter (better/worse): worse file deleted."

If you do a mass action from thumbnail right-click, it says this variant:

"Deleted from duplicate action on Media Page (action): one/both deleted."

180b1e  No.12238

Just did another clean up as hydrus was eating 6gb on launch and dups were starting to lag the program to much, its now starting sub 1gb with next to nothing changed since the last time I did this which got it down to 2.3~gb

still honestly see no reason for the watchers once they 404 to eat that much space, but good to see the program is getting even better with ram management.


as far as the images themselves go, yea, they are likely changing 1 pixel and re saving it, my shotgun approach to downloading has managed to seemingly sweep up everyone ever made but there are a few times where I knew the image was never getting posted again I just straight deleted the shitposts, but this time…

I thought I got rid of all of them, but apparently it's somehow finding more.


after a while I thought this too as I was getting somewhere of 455k images, it was really odd that there would be 3 or 4 images that were dups of the same image that would come in.

180b1e  No.12239


I have to ask, is it possible to have a custom sup finding thing? sadly I went though these a number of weeks ago, but I was checking an import archive, and apparently what I used to download corrupted images, it's more or less the standard at some point the image below this line is no longer loaded and its solid grey lines.

with images like this, would it be possible to have a 'corrupt dup' setting where you either point out where the image is good, or where the image is bad (a fill tool may deal with bad better then good making it easier to do) and have it discount that section from a dup search? my logic here is if I have a large enough archive, or the interests see me downloading similar images regularly, there is a good chance I have the un corrupt version somewhere else already

and just to head this off, these are images that have gone from a 120gb maxtor, to a 1.5tb seagate and survived 3 hdd failures there, to be moved to a 4tb and finally to am 8tb hdd, and finally to a currently 3tb image archive drive. odds are one of the drives fucked with the files in some way, or it was my horrendous method of acquisition all those years ago so the issue isnt my hdd corrupting images, its more I have corrupt images that may have to much corruption to be dup found

1cdbfc  No.12328


No, I am afraid sectioning off good and bad areas of images is way too complicated for my current system to deal with. I encourage you to delete all corrupt files.

File: 7a56ef4350b0bb9⋯.jpg (64.84 KB, 445x488, 445:488, 7a56ef4350b0bb96c5b6a4eb80….jpg)

ff2c17  No.11542[Reply]


92 posts and 41 image replies omitted. Click reply to view.

ff2c17  No.12253



I am sorry you had a hard time with this, but I am glad you are putting things back together.

I know it sucks to hear right now, but this sort of problem can be a great driver to get a good backup routine going. It happened to me about ten years ago, and has happened to more than a few hydrus users since I started doing this–you lose a load of shit and previous effort and decide to spend a bit of cash to avoid that pain again. I highly recommend a cheap external usb drive, either from a store or second-hand from a friend. Even just having a copy of the .db files–rather than including all your client_files as well–can make a real difference.

If you are interested and not sure how to do it:


ff2c17  No.12254


Thank you for this report. Have you had any crashes or power outages recently? The file identifier error may have occured due to a recent-ish hard shutdown of the client. As a precaution, I recommend you shut down the client and check the 'help my db is broke.txt' document under install_dir/db/ as background reading. You might like to just check the integrity of your db and maybe run a chkdsk to be sure your drive is ok.

Can you also check install_dir/db/client - 2019-4.log? If you scroll way down to the timestamp where this was happening (ctrl+f -> 'file identifier' will probably get you to it too), are there any error tracebacks related to the pause/unpause or missing/broken file imports? I would be interested in seeing that information, as it may better explain why subs were acting wrong here.

Subs may take a few seconds to register a 'pause' event, and will typically not stop working until the current gallery page or file being worked on is done, but they shouldn't flip behaviour entirely.

If this was just a blip due to a power cut or whatever, and you are confident your drive is healthy and database is not corrupt, you I think have two paths: if you keep using your client and it seems totally fine, especially if those subscription files are reimported and are fine, I think it may have fixed itself, but if you keep getting it, particularly in relation to the same set of files that are tagged on the PTR, your solution is to go services->review services->remote->tags->PTR and hit reset processing cache.

Let me know how you get on!

Post last edited at

ff2c17  No.12255


Yeah, it looks like they started putting secondary post attachments to a new subfolder in the URL structure, and it was being recognised and incorrectly truncated hydrus-side. I am rolling out a new url class for 348 that seems to fix it. Please let me know if you still have trouble here.

It wasn't a block, but the accidental hydrus truncation was doing something like this:




1f987d  No.12271

I've noticed that the url import isn't attaching tags to gelbooru and pixiv links anymore. Is there a setting I'm missing? I am on version 347 now, I was on 343 before.

ff2c17  No.12310


It seems ok to me in a drag-and-drop test on a gelbooru URL I just did now, and I do not think I have changed anything in the past four weeks–can you say any more about your situation? How are you importing this URL: through a search or a drag and drop? Have you imported any new parsers for those domains? And what are your 'default tag import options' set up as under network->downloaders->manage default tag import options? Do you have one TIO for all domains to rely on, or specific ones for gelb and pixiv? Do they all look correct?

File: ea31ab670c8dee7⋯.png (12.55 KB, 340x175, 68:35, mariadb-usa-inc.png)

928501  No.9068[Reply]

ITT: create proposals for making Hydrus more optimized.

Proposal: Why can't Hydrus switch to MariaDB?

If it is faster, then it should be better. The only trouble is having the need to rewrite the queries, which from an SQL standpoint should be a non-issue, right?

List of Databases with Open Source License and Open Source APIs:

SQLite - Currently used in Hydrus, has minimal features

MySQL - A more well-rounded SQL Database with user management

PostgreSQL - An SQL with complex features with less performance

MariaDB - SQL/NoSQL database with heavy optimizations

ElasticSearch - A literal search engine instead of a normal Database

Teradata - IDK



49 posts and 3 image replies omitted. Click reply to view.

138bc8  No.11586

55366b  No.11927

b2bc38  No.12295

f049d5  No.12302


Why don't you actually develop something on your own instead of endlessly shitting out github links

b2bc38  No.12307


Nah that is for >>12277

File: b2474c1d0a864f1⋯.jpg (193.25 KB, 1280x853, 1280:853, b2474c1d0a864f1d89b2622611….jpg)

9923a1  No.12279[Reply]

I had a good week, mostly working on small jobs and fixes. File and thumbnail access should be less laggy, some drawing code should be more resource efficient and stable, and the client api has a bit of work as well.

The release should be as normal tomorrow.

File: d7e685360192f2a⋯.png (20.52 KB, 722x449, 722:449, api-image-for-blog.png)

b45f6a  No.11626[Reply]

ITT: We propose new features that can be solved by using the API, and recommend new API commends for it

1 post omitted. Click reply to view.

59ad1a  No.11761

Here are some features that I can think of that are useful

1. Image tag and metadata "hash"

Image data hash, so if you were to download metadata from the same file twice, and the metadata has not changed, the hash would be the same, telling the user to not download it again and waste precious bandwidth.


IPFS operational API, so that you can create folder or file hash of any search results, and be able to pin or unpin them at will. This will be useful in the long term when people use it for Hydrus-2-Hydrus (H2H) P2P file share.

3. De-duplication API

internal APIs for file de-duplication software like Pippy360 for images and Chromaprint for music, and then allow external APIs be called for IQDB-like file search and hash download capabilities. (requires APIs for uploads first)

4. Headless Drone API

Headless "master+drone" Hydrus APIs, that allows one "master" to distribute downloader scripts and "download orders" to them, and the "drones" will return either downloaded images and tags, or an error message, back to the "master"

59ad1a  No.11764

Also having an API similar to this would be useful https://github.com/mserajnik/hydrusrv/blob/master/API.md

d9f85b  No.11802

d9f85b  No.11843


This is a one-upping https://github.com/rr-/szurubooru/blob/master/API.md now THIS is an API that we can learn from

b45f6a  No.12269

bumping this so that we can discover more uses of API better.

YouTube embed. Click thumbnail to play.

641234  No.12084[Reply]


zip: https://github.com/hydrusnetwork/hydrus/releases/download/v346/Hydrus.Network.346.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v346/Hydrus.Network.346.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v346/Hydrus.Network.346.-.OS.X.-.App.dmg


tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v346/Hydrus.Network.346.-.Linux.-.Executable.tar.gz


tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v346.tar.gz

I had a great week. There's a bunch of fixes and thumbnail improvements, a more powerful manage urls dialog, and a prototype OR search to try out.


Last week's big thumbnail change seems to have gone fairly well overall. There do not seem to be any huge errors anywhere, and the simpler file system and code is great, but I was annoyed by how slow certain regeneration scenarios turned out to be, and how some thumbnail resizes were a bit blurry.

To address this, this week I have cleaned and improved a bunch of thumbnail loading code. It was much nicer to do now I only have one thumbnail file to juggle. Thumbnails should now be aPost too long. Click here to view the full text.

21 posts omitted. Click reply to view.

a0ba9a  No.12145

> Can you give a concrete example of an export search that is failing for you? Is it by chance a 'system:rating' only search?

I have two exports, both are a simple search for one tag.

One to sync tag "bg", another to sync the tag "potential bg".

Both export folders have been left empty or incomplete when I use hydrus. I have to poke around the export settings to make sure they actually worked before closing hydrus.

460898  No.12146


>In 347, the 'evangelion' search will be run first, so the potential search domain of '-rei' is cut down from your entire collection to your x eva files.

Oh nice, especially with various system tags in place to limit the search, that'll cut down the time quite a lot for me.

On another topic, for the list of next big things, was the booru something you had in mind for working on? I had an idea about how it could be integrated more directly into the main UI, which I could write more about provided that there's interest in the booru right now.

641234  No.12250


I would like to have that, particularly for my own use, but unfortunately this functionality does not seem to be built in to my ui library, and furthermore my thumbnail panel is all my own custom code. I will have to write both ends of a system like this, so I have it as a 'big job' for now. It is currently "Write some ui to allow selecting thumbnails with a dragged bounding box" here: >>12152

641234  No.12251


Thank you, the 'delete after export run' bug should be fixed in 347 for export folders that make filenames that include subdirectories. Please let me know if you still get it in the new version.

641234  No.12252


Yeah, now here as a new 'big job' to work on: >>12152 . However, I am not totally sure what to do with the current booru. It was always a prototype that then never got revisited. I think next step is probably to convert it to the nicer new Client API code just so it is all operating on the same system, and then work on templating so users have more CSS/HTML customisation options.

Then again, I do know of some users working on booru wrappers for the Client API as-is, so if they can do a better job than I can fit in, it may be better ultimately for me to abandon it. If the booru does not prove all that popular in the poll and some other users can fill in the gap, perhaps we can revisit the whole idea and instead I give more power to them via the Client API.

If you have particular ideas on integrating the booru into the actual client ui, I am interested to know what you are thinking of. Please post in that big poll thread if you would like.

File: 7ece33b770e9701⋯.gif (565.95 KB, 300x166, 150:83, 7ece33b770e9701da946ffa95c….gif)

89c662  No.12143[Reply]

I had a good week. I fixed some bugs, extended thumbnail resizing into a much smoother two-stage system, and fleshed out the OR search workflow as I had hoped.

The release should be as normal tomorrow.

1a0602  No.12148

What 'bout wildcards in parent tags?

89c662  No.12244


That is an interesting idea. It might be dangerous if I were not careful, but I like it. Unfortunately, the tag siblings and parents system needs significant overhaul at the database level before I can do anything clever and new with it. If you are interested in new sibling and parent tools, please vote for "Improve tag siblings/parents and tag 'censorship'" here >>12152 .

File: d49758fd3e9ccfe⋯.png (54.11 KB, 2105x826, 2105:826, logo.png)

4ab0be  No.11035[Reply]

I was thinking since Pixiv has such a weird, complicated, and sometimes awfully designed site, us Pixiv bros should help each other out on figuring out ways to make it easier to use with Hrdrus. Post any tips, scripts, regexes, setups, etc.

7 posts and 2 image replies omitted. Click reply to view.

323f16  No.11786




Not just that, but "artists' favorite tags" and "tags' main users", from that we can create a network graph for better tag and artist discovery

4ab0be  No.11787


Those are nice scripts but my main issue is I need to keep turning new Japaneses tags into English using sibling. It just seems like such a clutter to keep turning Japaneses tags into English when they are already translated in that >>11759 2nd pic, unless there's a better way to deal with this and I'm going something wrong.

16c720  No.12060


sauce for the first image?

f3b3ee  No.12127



able to post all images visible? I like what im seeing

2781f9  No.12131

Delete Post [ ]
Previous [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15]
| Catalog | Nerve Center | Cancer
[ / / / / / / / / / / / / / ] [ dir / agatha2 / caco / choroy / dempart / jenny / tingles / vril / wmafsex ]