[ / / / / / / / / / / / / / ] [ dir / 1piece / cafechan / flutter / hkon9 / hkpol / india / miku / sapphic ]

/hydrus/ - Hydrus Network

Bug reports, feature requests, and other discussion for the hydrus network.

Catalog   Archive

Winner of the 42nd Attention-Hungry Games
/ara/ - A Place for Mothercons.

Comment *
Password (Randomized for file and post deletion; you may also set your own.)
* = required field[▶ Show post options & limits]
Confused? See the FAQ.
(replaces files and can be used instead)

Allowed file types:jpg, jpeg, gif, png, webm, mp4, swf, pdf
Max filesize is 16 MB.
Max image dimensions are 15000 x 15000.
You may upload 5 per post.

New user? Start here ---> http://hydrusnetwork.github.io/hydrus/

Current to-do list has: 1,507 items

Current big job: finishing login and domain managers and overhauling the downloader

HookTube embed. Click on thumbnail to play.

ef715e No.9208[Reply]


zip: https://github.com/hydrusnetwork/hydrus/releases/download/v311/Hydrus.Network.311.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v311/Hydrus.Network.311.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v311/Hydrus.Network.311.-.OS.X.-.App.dmg

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v311/Hydrus.Network.311.-.OS.X.-.Extract.only.tar.gz


tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v311/Hydrus.Network.311.-.Linux.-.Executable.tar.gz


tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v311.tar.gz

I had a great couple of weeks. E3 was fun to watch, and then I got back to proper work, mostly fixes and improvements to the new download systems.

pixiv fixed and other downloader stuff

I have made a new parser for the new dynamic pixiv layout. It was not simple, but it seems to work ok, including for manga. It only gets unnamespace tags in romaji/kanji (fetching just romaji/translations was a bitPost too long. Click here to view the full text.

25 posts and 5 image replies omitted. Click reply to view.

78ef0c No.9257

File: 3117d7558d503e6⋯.png (357.96 KB, 1730x1699, 1730:1699, ACDSee8Pro_2018-06-24_01-3….png)

File: 98ffd4a727fda3a⋯.png (64.07 KB, 430x386, 215:193, ACDSee8Pro_2018-06-24_01-4….png)

File: 7d51689ae92bf32⋯.png (4.65 KB, 1073x71, 1073:71, client_2018-06-24_01-47-11.png)

File: 226c543721749c9⋯.png (4.82 KB, 1073x71, 1073:71, client_2018-06-24_01-53-31.png)


If you are going to put them in a dialouge window, it would have to be one where you can use the background parts of the program too, thinking about it a bit more, here is a program you may want to look into

acdsee 8 pro, this version of it specifically

note the area on the left above the thumbnail, you could have in the case of threadwatchers

the thread window with its basic functions,

a settings tab

and a tag tab

pretty much all giving you the same amount of information, just at different times.

that said thinking of it more,

clear highlight and highlight on all tabs would be very good, especially because of tabs like this one, 96000 with 14000 already in db, I would be able to unhighlight it, set the options to only show new ones, and then rehilight it, it seems quite a few issues I have would be solved with this being a part of every downloader.

On the note of download tabs, do you have any interest in expanding the multi watcher to a general multi downloader and multi gallery downloader?

The tab that has 96000 images is one from gelburro I believe, just looking at what was downloaded I think I put in 20 or so different artists in general, and then also added some tags like short stack or some character names, needless to say the things a gigantic clusterfuck of images.

however a multi gallery I could have each search handled on it own viewable selection,

and a multidownloader, well here, the 15000 image page is from a /aco/ download of old 404 threads.

Till you got the multithreadwatcher in place, what I did was I saved tabs, and I waited for them to either die or 404 then download them en mass, I said the first day multithreadwatcher came in that I had something along the liPost too long. Click here to view the full text.

File: 96339c942441720⋯.png (362.43 KB, 700x700, 1:1, confused_6.png)

cc0a1a No.6021[Reply]

Here you can ask questions so that the board is not clogged with small threads.

>>2231 has reached its bump limit, so I made a new thread.

743 posts and 150 image replies omitted. Click reply to view.

a2c666 No.9252


>throwing the new version files over your already existing hydrus folder fuck it up

Pretty sure not. I've edited files and reloaded them back etc. If you load the same file into hydrus it will just replace the old one I think

5a7b47 No.9253

Ok, maybe I'm just retarded, but I can't make my pixiv subscriptions to download manga/album with the new version.

Is there something else I need to do on the subscription or another config?

d76cbb No.9254


Did you set up your pixiv account in network > manage pixiv account?

5a7b47 No.9256


Yes. My subscriptions work, but only download "single" files.

For example: On this user https://www.pixiv.net/member.php?id=549226 it downloaded everything on the illustration category but not a single file of the Manga.

c1b562 No.9258

File: bbe4f1d2122d318⋯.jpg (323.69 KB, 582x787, 582:787, bbe4f1d2122d31835ba1f53eb3….jpg)

I have a whole pile of files tagged with their pixiv ids. Is there any way to turn all these into "known urls" beyond manually converting them?

HookTube embed. Click on thumbnail to play.

2da50d No.9062[Reply]


zip: https://github.com/hydrusnetwork/hydrus/releases/download/v310/Hydrus.Network.310.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v310/Hydrus.Network.310.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v310/Hydrus.Network.310.-.OS.X.-.App.dmg

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v310/Hydrus.Network.310.-.OS.X.-.Extract.only.tar.gz


tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v310/Hydrus.Network.310.-.Linux.-.Executable.tar.gz


tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v310.tar.gz

I had a productive week. A bunch of stuff is fixed, and I finished a first version of multi-file url downloading.

multi-file support

A limitation of the old download system is that it always assumed a single 'post URL' could only ever produce a single file. This week, I have added full support for multi-files-per-post-URL to the new downloader. If a new parser generates multiple 'downloadable' URLPost too long. Click here to view the full text.

30 posts and 15 image replies omitted. Click reply to view.

2da50d No.9176


That's a breddy hefty client you have there, btw. What's the general start-up time, and how much memory does it use just sitting there? What's the idle CPU usage like?

161b8a No.9177

I'm pretty new to Hydrus (using it for about a month) and just wanted to say thank you for pouring so much time and effort into this.

dab996 No.9193


working through the images at the moment should get it down soonish but I procrastinate.

ramuse is 9-12gb, cpu idle is around 0-4% but i'm on a ryzen 1700

startup time, a few versions ago start up got significantly faster, but it will hang anywhere from 5 minutes to 15 minutes before its useful,

the bigger problem is the hangs every 20-40 seconds that last around 10~ seconds.

honestly, if you are able to get a highlight and clear highlight option working for all tabs, I think this kind of a session would be childsplay.

ef5bc7 No.9196

hallo where are the custom downloader engine :::—–DDD

2da50d No.9247


Great, I am glad you like it. Let me know if anything doesn't work for you. If you feel comfortable, I'd appreciate knowing what you have found difficult and easy to pick up. Improving the help is a constant job, and feedback from new users is always helpful.


I am pleased the CPU is so low! I may be getting that ram down too in the nearish future.

I am not sure what the hang is about, but if you turn on help->debug->profile modes->db profile mode/pubsub profile mode and wait for the delay to hit, we might catch it. Be warned with your busy client these modes may spam your ui into oblivion, so make sure you hit gui actions->save last session gui session beforehand, just in case it crashes/locks everything up. The profile modes will make a new appropriately named .log file in your install_dir/db directory. Please send that to me through email or discord DM or some other private means (it'll probably have some semi-private info in it), and I'll see what is hanging things up.

For 'clear highlight for all tabs', do you mean, say, a menu entry up top that when clicked will clear the highlighs for all current multi-watchers?


Getting there m8. With pixiv working, the Post URL parsing part is basically done now. Now I'm moving on to Gallery URLs and then a 'Searcher' object that'll tie it all together. If you want to check out some early help, please go here:


File: 1426721772716.png (100.78 KB, 1624x1081, 1624:1081, 1327614072601.png)

7f2c0e No.471[Reply]

Drag and drop windows with tag rules. Show two windows side by side and one window can be programmed with the rule "ADD tag foo" and the other one has the rule "REMOVE tag foo, ADD tag bar" and you can drag and drop files to them.

Deriving tags from regex of other tags/namespace tags. A file has the tag "filename:big_ugly_name" and we could regex that namespace for another tag.

Tag sets with hotkeys: save a set of tags under a hotkey so it's quick to add them to a file while filtering

Opaque window behind tag list in the corner so it doesn't get hidden by picture background

Option to default certain mime types to be excluded from slideshow and only open externally, will help with videos with odd codecs that don't preview in the slideshow correctly

Option to specify hamming distance in "find similar images", you can't change the option once it's in the filter window and you have to enter the hash manually in the "system:similar to" option
560 posts and 186 image replies omitted. Click reply to view.

486cfb No.8990

Youtube, Bitchute and PewTube subscription support with multi-resolution

9908ab No.9124

Proquint as a replacement for Hex/Base64 for SHA256

cb7da8 No.9223

web UI/http POST/CLI so that I can download shit from afar via SSH/webpage

23e98b No.9244

Hello, is it possible to choose the browser that Hydrus opens links in? Rather than just using my default browser in Windows

ddb9b3 No.9246

File: e287b8d099f9358⋯.png (551.51 KB, 1920x1080, 16:9, ClipboardImage.png)

How about a "tag in order" option?

I've been downloading mangos from threads and it's a pain in the ass to tag pages manually, a generic way to tag images in the order they are like

Select images -> right click -> tag in order -> incremental -> "page:1"

and it would take the last number and increment it by one each pic, so the first image would be page:1, the second page:2 and so on.

File: 2b5572a7158177b⋯.png (18.71 KB, 418x273, 418:273, 2018-06-09 13:13:28.png)

File: bc7af941c2be31f⋯.png (47.1 KB, 516x524, 129:131, 2018-06-09 13:14:13.png)

08f327 No.9091[Reply]

I installed Hydrus from the AUR package, i should have known better i won't do it again i promise ;-; and after updating to version 309 (i wasn't able to figure out which version I had before that but i'll try if it's still relevant after the next part) I got

Gdk-Message: 13:28:54.154: client.pyw: Fatal IO error 11 (Resource temporarily unavailable) on X server :0.

when running hydrus-client in the terminal. If I move the db folder so it starts a new one, it's perfectly fine. I tried downloading a a fresh copy of v310 from github and running it with my db folder, and I got these errors instead. Can this be fixed?

dc3433 No.9150

Thank you for this report. This is very useful information. I apologise for the late reply–I am just coming off a light E3 week where I did not keep up with my messages well.

I believe the 'resource temporarily unavailable' error may be due to that client being unable to show that 'can't repair this db' error in certain situations. I will make sure to fix how that popup displays.

As for the local_hashes issue, I am not sure what has happened here. My best guess is you maybe updated from pre-v262 and that update applied badly? Or did you update from just, say, v306?

If this table disappeared in just a simple, fairly recent update, you may have had a hard drive fault, and you probably want to check out the 'help my db is broke.txt' file in your install_dir/db folder as background reading. Losing local_hashes isn't the end of the world, but if your hard drive is three months away from dying, you want to check and sort that out now.

If this is a big update that went bad, there is less to worry about. My code maybe messed up due to bitrot, so we'll just have to fix it retroactively.

In either case, to get your client booting again, we'll need to create that table manually. Make sure your client is all shut down and then go to your install_dir/db directory. Run the sqlite3 executable, which will bring up a sqlite shell prompt, and then paste this in (you should be able to do it line by line):

WARNING EDIT I JUST THOUGHT OF: Do this first, please:

.open client.db
SELECT name FROM sqlite_master WHERE type="table" AND name="local_hashes";
(does it give you 'local_hashes' result or no result at all after this line?)

If you still have a local_hashes in your main db, then we can just move it. This preferable to the following solution, so let me know and I'll write you a different fix. Otherwise, open a new sqlite prompt and:

Post too long. Click here to view the full text.

08f327 No.9230

File: dbeab90877afc06⋯.png (5.58 KB, 349x50, 349:50, 2018-06-21 21:58:32.png)

File: 5c77b0803604b2e⋯.png (1.48 MB, 1600x1151, 1600:1151, hydrus client 2018-06-21 2….png)


>My best guess is you maybe updated from pre-v262 and that update applied badly?

Apparently, that's all it was. I got pic related after running all your commands (i somehow managed to miss what you said in between them and blindly ran them all) which eventually lead to

>OperationalError: table local_hashes already exists

But after restoring a backup (i backed up the whole supposedly-broken db folder when it first failed to load) and downloading version 297 from github, it updated perfectly fine and I was able to update to 311 from there. Apparently my current desktop theme doesn't go so well with Hydrus, but whatever. Thanks for the help!

>be warned, this will ultimately be a big job

for you

File: 3b0eb8179351430⋯.png (2.07 KB, 227x38, 227:38, chrome_2018-06-14_02-10-08.png)

c3d72f No.9127[Reply]

wow the perfect image grabber but its out of reach.

upload to a mirror you neanderthal

4 posts and 2 image replies omitted. Click reply to view.

733516 No.9136

File: 85a716cef1db0df⋯.png (11.33 KB, 300x300, 1:1, please.png)

a0442a No.9137

File: ccec7e4c64ec90e⋯.png (36.83 KB, 726x236, 363:118, Untitled.png)


about $30 a month

eed594 No.9184


what the fuck?

I'm australia guy btw

I get like 50 down here, but github speeds start slow and grow and speed up on a linear scale.

Application used to be distributed from a dropbox, but as far as this goes, github's peering is just utter shit

866c45 No.9185


Anon he was being Australian. That'd how we speak.

a359ad No.9221


In the future version of Hydrus, you can VPS to any of your favorite hosts (even your friend in a foreign country) and they will send you the HDD once it is done "doing its job".

But until then, you will have to do everything by hand.

File: afce1fb7c68c55e⋯.png (1.52 MB, 1280x1732, 320:433, afce1fb7c68c55e7d5ade0f1e7….png)

File: 5f505978378ccd3⋯.png (7.83 MB, 3400x4600, 17:23, 5f505978378ccd323ed5e04eec….png)

File: 1125a234c31ad6d⋯.png (227.42 KB, 1939x557, 1939:557, client_2018-06-01_18-03-28.png)

9b85e6 No.9010[Reply]

I think that suggestions make the program better, improvements here and there also go a long way, but really there are hard aspects of the program that cause people to stop everything they are doing till they are improved/reworked.

let me give an example. I have an image db that is currently 2 million images large.

There was no way in fuck that I was keeping tabs on this shit on its own, so the program is very good in that regard, at least my shit is sorted and can be brought up if I tag/rate something.

now, here comes the stop gap.

I have 2 million files, and before my db corrupted, I had something around the area of 200k potential matches for duplicates

so I need to go through the duplicates first

my database broke through so I have only 827k images in the dup finder with 73k images.

Now thinking ok, while things get sorted out, why not do them? dup finder only works on 2 images at a time, regardless of if the rest of my db in or not there is work that can be done

well here is a random pair

im on a 4k monitor, a 55 inch 4k, so when an image is small and one is big, you can generally see one is better easily, however the smaller one is not worse by a lot, note the bigger ones embers, on the yellow shit at the bottom right, zooming into 100% on it makes it clear its not just an upscale, and that it actually is a higher resolution. but if you put these in the program, and find these in dup detector, note that due to the slightly different resolution the program won't keep zoom or where its zoomed in between the two images.

also note that trying to move where its zoomed into fast with the mouse results in making a selection rather then moving the image.

I also noted at least with ratings a while back making a 'this one is better/prefer this one' selection did not transfePost too long. Click here to view the full text.

20 posts and 5 image replies omitted. Click reply to view.

e05b0e No.9144


Updated to 310 but same behavior. I should note that it's ratings that are broken. If I use system:limit and punch in a tag then it brings them all up fine. but without ratings there's no real order to it. I could tag with 'favorite' but thats just redundant to what ive already been tagging with favorite ratings.

ad534e No.9188




I don't know if this is "working as intended' but all stuttering I had cleared itself when I would routinely go to "undo -> closed pages -> clear all"

I would start to notice this behavior after multiple queries of multiple 10k+ files, so it became part of my routine.

9b85e6 No.9192


Honestly it sounds like you want

limit images and system favorite but you are only ever doing one at a time.

you know you can put as many filters as you want into it? at least as far as I have looked as many as you want.

81bbf9 No.9216

The only thing that I want from Hydrus right now is subscriptions to Reddit, since I get a lot of images from specific subs.

c6d35b No.9219

tl;dr but I agree that duplicate sorting is incredibly tedious right now. I realize the problem is that an image could be detected as a potential duplicate for two different images not within the similarity threshold to each other, but the "show some pairs" function works fine so it would be really nice if the dupe sorter could bring up groups like that to scroll through, or at least work through one group like that in pairs rather than swapping between different incomplete similarity judgments like it does now.

I'm usually using a 1366x768 screen but I still would like the option to scroll through all duplicates that would otherwise appear using the "show some" button, since I can't easily do the same dupe operations from just the normal window without launching the dupe manager popup.


I'm mostly waiting on the new download manager where we can make custom user scripts for various sites, because not only could I then use it for subreddits and "scrape through" to linked imgur stuff, I could scrape links and comments and hopefully also scrape through to video posts people make from other sites.

Actually, maybe what Hydrus really could use is a link scraper so if there's unscrapeable content it dumps it into a txt file that it keeps in its db so I can go back later and grab the things it couldn't get….and/or run another custom userscript download on a list of URLs from a given site.

File: ea31ab670c8dee7⋯.png (12.55 KB, 340x175, 68:35, mariadb-usa-inc.png)

928501 No.9068[Reply]

ITT: create proposals for making Hydrus more optimized.

Proposal: Why can't Hydrus switch to MariaDB?

If it is faster, then it should be better. The only trouble is having the need to rewrite the queries, which from an SQL standpoint should be a non-issue, right?

List of Databases with Open Source License and Open Source APIs:

SQLite - Currently used in Hydrus, has minimal features

MySQL - A more well-rounded SQL Database with user management

PostgreSQL - An SQL with complex features with less performance

MariaDB - SQL/NoSQL database with heavy optimizations

ElasticSearch - A literal search engine instead of a normal Database

Teradata - IDK



10 posts and 1 image reply omitted. Click reply to view.

4e2f99 No.9121

I would not mind runing mariadb daemon for hydrus.

In fact, i am running one right now, and it would be great if i could set hydrus up to just connect to an existing database.

0a095d No.9123


What about file system parities? Would installing Hydrus on FreeNAS with ZFS be a good idea? What about Linux with BTRFS?

3e2c75 No.9182

3e2c75 No.9183






Some of the points:

1. 8GB of RAM minimum, 12GB minimum if using plugins or jails, 1GB RAM per 1TB (conservative) or 3TB (liberal)

2. Don't use RAID controllers, just use Hot Bus Adapters to connect the drives to the motherboard (software "RAID")

3. FreeNAS needs bare metal, NOT VMs (but putting plugins or jails into FreeNAS is a good idea)

4. Intel CPU has more support than AMD, and LSI has the best Hot Bus Adapters (Marvell and J-Micron is okay)

5. 7200 RPM SAS or Enterprise SATA will work as HDD, do not use desktop drives for this to prevent IO errors

6. RAIDZ1 is like RAID 5, RAIDZ2 is like Z6, RAIDZ3 has triple parity, each vdev/group only has one-drive speeds

7. "ZFS intent log" should be on RAM (and on power-protected SSD if you wish), without it the whole vdev would fail

928501 No.9217

https://ponyorm.com/ can actually simplify SQL queries into something more python-friendly.

File: 031bf08c45f341f⋯.gif (743.87 KB, 478x384, 239:192, 031bf08c45f341f7ec1f10569c….gif)

4dd9a8 No.9191[Reply]

I had a great couple of weeks. E3 was fun to watch, and then I got back to proper work. I managed to fix the pixiv parser for their new layout–including for manga–added several bells and whistles to the multiple watcher, and sped up some search code.

The release should be as normal tomorrow.

7a0119 No.9195


>I managed to fix the pixiv parser for their new layout–including for manga–added several bells and whistles to the multiple watcher, and sped up some search code.

G-God, is that you?

949a26 No.9202

>I managed to fix the pixiv parser for their new layout–including for manga

So will it try to redo the pixiv manga links in my subs or should I reset those manually? Is there a another option besides "retry failures?" Since those aren't exactly failures

4dd9a8 No.9209

File: 3e138883bacbe78⋯.png (951.45 KB, 1022x874, 511:437, 3e138883bacbe788a8ce537f94….png)


I'm no one special, just an Anon. Let me know if it doesn't work for anything.


Yeah, please try a few this week by going into the sub file import list and selecting a few 'ignored due to manga' entries and right-click->try again. Let me know if there are any big problems, and if it all looks good, I'll add a big 'retry ignored' button to the subs dialog so we can do it en masse.

File: ab23431121b8563⋯.jpg (96.69 KB, 492x650, 246:325, ab23431121b8563a7bc3b7ac57….jpg)

0675b5 No.5115[Reply]

Gonna start another thread because >>173 is becoming too big for most people.

In the tag manager, whenever multiple tags are selected for multiple files with different tags, if you press the del key, the program will still ask you if you want to delete them all or put them all in the selected files. This does not make sense, if I'm pressing the del key it's obvious that I want them gone.

435 posts and 133 image replies omitted. Click reply to view.

06c25c No.9059

File: 99ae99792749bb7⋯.gif (593.76 KB, 500x281, 500:281, tumblr_ohhjgkh1fy1r687oqo3….gif)


>json_dumps, dump_type 22, json key 'temp_path_override'

Fixed, easy peasy. Apparently having the temp file be in /programs is a bad idea (I hecked it up while trying to hook my browser, guaranteed). I can now open hydrus again! Thanks!

85b896 No.9081

Hydrus crashes when attempting to import images and folders, tagged or not. Happens when pressing the "import" button, or its counterpart in the tag manager.

Error message is:

> corrupted_size vs. prev_size

> terminated by signal SIGABRT (Abort)

Running Archlinux 64-bit, with mostly updated packages.

85b896 No.9110


Also, another bug I've encountered is

>client: Fatal IO error 11 (Resource temporarily unavailable) on X server :0.0.

Triggered when I'm in viewer mode. Same specs. According to https://stackoverflow.com/questions/25790890/xio-fatal-io-error-11 it may be because of running a 32-bit application on 64-bit, but I'm not sure.

I'll attempt a manual compile, to see if it fixes any of the issues.

35b3df No.9163

File: ba10497cd6a5f08⋯.jpg (1.93 MB, 1456x2929, 1456:2929, ba10497cd6a5f085c021c4ebc6….jpg)


Great, let me know if you run into any more trouble.



Thank you for this information. I am still working on improving Linux stability, although these last problems are proving tricky to pin down. Some may be Arch-specific or -predominate, so I would appreciate any more information you discover.

This corrupted_size vs prev_size is a new one. I'll have another look at how those import prep dialogs are converted to an hdd import object.

For hydrus, the resource temp unavailable is almost always a 'talking to a gui object from a non-main-gui thread' problem. The old version of wx was very forgiving on this front, and even when it had a problem, it usually just through an assertion exception that I mostly suppressed. Unfortunately, this new version–while it is great in a bunch of other ways–gets very pissy, particularly with gtk on Linux, if I even touch my own custom variables in gui object from the wrong thread. I know the fix, but chasing these last occurences down has not been simple, particularly because the crash often only triggers 30-3600 seconds later, when the object is eventually garbage collected. The situation is still a bit shit, but improving.

I am going to keep working on this. I'm also crossing my fingers that the next wx update has some better safe-handling at the C++ level. I suspect some of the extreme sensitivity is unintentional.

1a08f3 No.9180

There's no sane way to install python3-wxgtk4.0 on a Debian 9 machine, installing python3-wxgtk4.0 would mean basicially mean upgrading to sid. here's a List of packages that would be removed/installed/upgraded


I guess the only sane way to use hydrus on Debian 9 would be docker…

But I'm too stupid to hack together such an Image

File: f7061c9177b1b35⋯.gif (144.02 KB, 300x300, 1:1, f7061c9177b1b35d11ed034276….gif)

b71802 No.9115[Reply]

As planned, I took this week easy for E3. While I did get a little neat work done, there is not enough to justify a release, so there will be no release tomorrow.

I'll do a bit of regular work tomorrow, and then I am back to normal schedule, meaning v311 will be on the 20th.

8 posts and 5 image replies omitted. Click reply to view.

b71802 No.9164

File: e7f7b0e9a9b4469⋯.jpg (473.41 KB, 1400x979, 1400:979, e7f7b0e9a9b4469d90cfe49a91….jpg)



Thanks for this info. This looks great stuff–if it is all correct with you, I'll point people at your repos here in the v311 release post if they would like to experiment. I am interested in knowing further how these different systems work well and badly.

For the one-image-at-a-time issue, it has long been a thought to convert the file-lookup system to an automated 'figure out any tags for these 10,000 files in idle time please' maintenance routine, particularly once I had introduced the new bandwidth system, which will help us not DDoS the services we are pulling from. Your use case here is another vote in doing something like this, and generally to generalise the way(s) hydrus can pull tag recommendations from other services and scripts.

I'll reiterate that I can't put any time into this atm, but I am interested in having an ongoing conversation about how I can make these workflows better for you. I am keen to get hydrus and The Imageboard Community into the ML game in the coming years.


Yes. In making hydrus, I have come to make a point about putting 'human eyes' in front of certain decisions, especially for large automated systems. Figuring out a good workflow is often as difficult as getting the technical side working. It is easy to fuck up a script, and if that script touches 100,000 files, it can be a huge pain to fix. Tag siblings is a good example of this–it seemed simple when I got in, but it turns out for multiple reasons–human preference and complexity of language and translation issues and simple human mistake and technical complexity at the data and gui levels and low CPU availability at certain critical moments in the sibling processing pipeline–to be much more complicated.

If we gain the ability to harvest millions of new tagPost too long. Click here to view the full text.

7927a2 No.9171


>if it is all correct with you

Sure! I think the code in that repo is currently workable enough for some user testing. I'll check back if you do so and try to debug any issues that come up.

>I can't put any time into this atm

Don't worry, stick to your schedule. I'm happy to help whenever you're ready.

Though I don't check 8ch much, so if you're trying to find me try pinging my github or this email.

>"Here is a 'feminine penis' detector."

I want to build this just to freak out the rest of the world


>would you be able to set auto tagging tags to their own category?

One solution would be to assign the auto-tags to their own namespace, which I think is possible from the parsing script menus.

>it gives things that are completely unsafe nsfw status, and things that are safe nsfw

Yep, I cherry picked an image where it did decently for the example. "shameimaru aya" is an i2v tag but it missed it in >>9151, and there were a few where it incorrectly guessed safe/questionable/explicit (about 5%).

I'll admit that my ML experience is primarily academic / personal, so it's pretty interesting to see what some real user-facing requirements look like. I'm pretty sure everything you mention is possible, but I'd have to do some reading to tweak the standard process to make it fit (except export to png, unless you like 200MB png's).

I would like to build that "nntag" module, and I would like to make it so that tagging models can be shared, hot-swapped, and trained on a distributed network. Whenever you're ready to think about this stuff, hit me at either of the contacts above and we can talk about how to make it work.

65c0d3 No.9172

yo hydev, i think the URL downloader broke. im copypasting links from 8chan itself and they give me a "could not find parser for 8chan file URL class" error

b71802 No.9174




Thank you for this report. 8chan direct file urls have a 'file' url class, and the url downloader is lumping them in with parsable url classes rather than just trying to download. I will make sure to have this fixed for v311.

For now, pasting one of the alternate 'file_dl' links, like the (h) or (u) ones for 8ch, should work ok.

65c0d3 No.9179


cool, have a nice week hydev

File: 2a1705f785b8ca0⋯.png (3.36 KB, 512x110, 256:55, sc.png)

File: aee9a6fb7a049fd⋯.png (3.25 KB, 512x109, 512:109, vidyart.png)

File: f6c617a7c8b52f3⋯.png (2.87 KB, 512x85, 512:85, xxx.png)

16e49a No.9082[Reply]

Don't think there's a thread for those and the script thread is a bit more hands-on, so there we go.

Grab 'em use 'em

dc28d0 No.9086

16e49a No.9089


I think booru.org subdomains can have different stylings, which would break the parser.

Feel free to try it though ! You can arbirarly assign parsers in hydrus to whatever website you want.

dc28d0 No.9098


Why not we build a "booru styling tester" to simplify the process? Out of all the links in the list many share the same default style.

File: 357054403825d94⋯.jpg (44.3 KB, 1024x576, 16:9, 1481168675293.jpg)

ead5a8 No.5187[Reply]

Hey everybody, this is just a thread to talk about our everyday life with Hydrus.

How are you coping with your file hoarding addiction now that you're being enabled?

Are you remembering to take some time to tag what you download?

You ARE making a monthly backup next week, right?

What kinds of files are you saving, and what sorts of tags are you committing to the repo?

Right now I'm sorting through the downloads from a few of my NSFW subscriptions pre-tagging. They are all about half stuff I want and half gay shit I don't want, so I put "impure" in their subscription tags and I just go through 100 at a time and purify them. It does tend to break about even with me keeping 50 files out of each 100. Pray for me anons.


105 posts and 34 image replies omitted. Click reply to view.

f449be No.8744

File: a00b17aaa48d304⋯.jpg (74.63 KB, 667x869, 667:869, 713cbd8002b49cf01a36790244….jpg)


>In my gross tech ignorance, I once day just felt I'd found a porn image I really, unironically liked. Wasn't sure how to sort it beyond my broadly-named ~5 porn folders. So I created a specific folder for it, and put a shortcut to the image in it. And that started my really inefficient sorting method.

Ok. So this post was me. It was actually horrifying converting this sorting method to Hydrus, since Hydrus couldn't fetch target images from a shortcut fed to it. I opened a folder full of shortcuts in Irfanview, but it saw nothing. My only recourse as far as I could tell at that point was to manually open each and every shortcut in a browser (my choice was Firefox,) and manually resave each and every image. Which was exactly what I did for the first 305 folders, and 9,652 files (shortcuts; I just highlighted everything then looked at Windows properties for this number). Until I asked for help in 4chan /g/'s /sqt/ thread, and an anon replied saying he'll take a crack at it. It was like that scene in Bleach, where that asshole character offered to save Rukia, but seconds later said he was joking, so Rukia lost resolve she never knew she still had, and she screamed in anguish on the spot. I basically didn't want to do this anymore after this anon showed up. I had 62 folders left, and 4,107 files (shortcuts) left, since I was going chronologically reversed order, so the older ones were more dense with sorting, and poor sorting.


Anon told me to run the following in powershell:


#must be run from admin prompt as it modified the registry
Set-ExecutionPolicy RemoteSigned;

And then run the following in powershell as well*:

(*note: "Set $simulate = $false when you want to Post too long. Click here to view the full text.

50ce99 No.8845


How do I run this in Hydrus, I see code but where would I plug it in? Sorry for not really understanding, I'm just trying to optimize my porn collection in every way possible.

1ebffc No.8885

File: 30bd489df94b2fc⋯.jpg (1.04 MB, 2500x1213, 2500:1213, 1507046780099.jpg)


in about 10 years if storage becomes huge and affordable we should attempt The Great Merge where you can opt in to starting a new database and have all your files and tags P2P distributed with everyone else

1ebffc No.8886


actually that's probably a bad idea since even if you can blacklist tags people will still manage to dox themselves

3ad3b7 No.9079

File: bc8d8d40e2ae954⋯.jpg (18.6 KB, 284x300, 71:75, downhill.jpg)

>need to tag 250 images from two and a half months ago

>keep getting distracted

>keep importing more files

>400 in inbox

I need to cut off my internet and sort this shit before I end up with thousands of images with only 1 or 2 tags

File: 75a1b729606c7bf⋯.gif (370.65 KB, 400x300, 4:3, 75a1b729606c7bf3a4fb013c24….gif)

6926d3 No.9056[Reply]

I had a productive week tying up some loose ends. There are a whole bunch of fixes and improved error handling, and I managed to get the first version of single-post-multiple-file download pipeline working. There are new parsers for twitter image posts, inkbunny multiple file posts, artstation posts (which can also have multiple files), and pixiv manga posts, which I think I just about have working right.

The release may be a bit late tomorrow, as I still have a lot of testing to do. Also, tomorrow's v310 will be the last release for two weeks–v311 will come on the 20th.

28b0b0 No.9060

Looking forward to trying the Pixiv manga parser! Enjoy your well-deserved two-week break, based dev.

HookTube embed. Click on thumbnail to play.

96555a No.8979[Reply]


zip: https://github.com/hydrusnetwork/hydrus/releases/download/v309/Hydrus.Network.309.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v309/Hydrus.Network.309.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v309/Hydrus.Network.309.-.OS.X.-.App.dmg

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v309/Hydrus.Network.309.-.OS.X.-.Extract.only.tar.gz


tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v309/Hydrus.Network.309.-.Linux.-.Executable.tar.gz


tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v309.tar.gz

I had a good week with a bunch of unusual work.

tumblr GDPR

When GDPR hit, tumblr introduced a click-through page for all European Union and European Economic Area users. Unfortunately, this page applies (likely unintentionally) to their old JSON API as well as regular pages, so it broke the current hydrus tumblr downloader.

I have writtPost too long. Click here to view the full text.

33 posts and 6 image replies omitted. Click reply to view.

726092 No.9050


Well, I did only just discover Hydrus, and I've only read the intro that's linked at the top…

Anyway, regardless, what I tried to mean when I said that quoted sentence is a true standard that everyone uses and connects with. Sorry if that wasn't clear. I already knew Hydrus and other platforms have databases that are community managed. I'll happily use Hydrus, but I hope for a standard in the future, which I think may probably only arise if OS vendors implement it and make it a standard way to interact with files, folders, and links (to arbitrary locations).

b2e314 No.9051


how do you propose this standard is implemented when every site has its own tags and most sites simply let you type anything as a tag?

what we have is probably the best we can get, contribute to it by not making shitty tags, and replacing shitty tags with the proper ones. You can petition tags to be removed in both PTRs and use "should be namespaced" as a reason for removal.

96555a No.9053

File: 9fecfb4eb4cbdcf⋯.jpg (429.74 KB, 1400x1000, 7:5, 9fecfb4eb4cbdcfd09dcb4ca43….jpg)


Thanks, inkbunny multiple files should all be fixed for tomorrow. I think the files may come randomly just out of order, like 2, 3, 4, 1, due to some unusual markup on their end. Let me know how it goes anyway.

Yeah, my old booru system takes advantage of very common markup across the big booru softwares. For DA and HF etc…, I wrote custom hardcoded solutions (I actually did these before boorus, I think, ages ago), but the whole thing proved way too difficult to maintain on my own. This new system will be much more flexible, allowing any user to create and share and maintain new downloaders for any 'we got pages of thumbs and pages of files' kind of gallery site without me having to fold it into an update.

BTW: While DA and HF tag search works on a more complicated system on their end than how boorus do a direct tag-map, they still spit out standardised chronological and stable results in a grid of thumbs, which is how I am able to do that. On my end, it is all just different ways of going "hey site, here's _search_text_, please give me some thumbs to parse'.







This big downloader overhaul is my attempt to do basically what you are talking about here. Maintaining hardcoded downloading solutions was eating up too much of my time, so I am now moving the whole system to a common user-editable standard. Unfortunately, it turns out that Post too long. Click here to view the full text.

726092 No.9054


Well I haven't really thought deeply about the tagging system itself, all I'm saying is that it would be great if there was a standard that everyone just uses.

But thinking about the tagging system, I think that using AI like I mentioned to help could be interesting in the future. It is already possible to use AI right now to identify with relatively good accuracy what is contained within an image, and it's only going to get better with time.

I also think it could be interesting to let tags themselves have the ability to be tagged, which itself would be inherent in the system I'd like to see happen (you can tag anything). I think it would also be important, if it isn't normal across these systems yet, to have the ability to easily access the properties of a tag, so that you can filter tags themselves beyond using the tags' tags. For instance, I may want to see all tags that count as scenery, but sort by the amount of entries each tag has, or sort by last date modified. The database should also probably be decentralized, to better manage the vast amounts of data that will be generated from everything being tagged. Now I guess this system could reduce the importance of tag accuracy, for better or worse, because you can filter tags themselves by tags. And so you should be able to have default filter settings on so you don't get results you don't want.

b2e314 No.9055

File: f6033aab2e1dd21⋯.png (99.78 KB, 248x247, 248:247, 1345228229506.png)


>tagging the tags

Delete Post [ ]
Previous [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15]
| Catalog | Nerve Center | Cancer
[ / / / / / / / / / / / / / ] [ dir / 1piece / cafechan / flutter / hkon9 / hkpol / india / miku / sapphic ]