[ / / / / / / / / / / / / / ] [ dir / abdl / animu / doomer / leftyb / lovelive / mde / tingles / zoo ]

/hydrus/ - Hydrus Network

Bug reports, feature requests, and other discussion for the hydrus network.

Catalog   Archive

Winner of the 68rd Attention-Hungry Games
/d/ - Home of Headswap and Detachable Girl Threads

January 2019 - 8chan Transparency Report
Comment *
Password (Randomized for file and post deletion; you may also set your own.)
* = required field[▶ Show post options & limits]
Confused? See the FAQ.
(replaces files and can be used instead)

Allowed file types:jpg, jpeg, gif, png, webm, mp4, swf, pdf
Max filesize is 16 MB.
Max image dimensions are 15000 x 15000.
You may upload 5 per post.

New user? Start here ---> http://hydrusnetwork.github.io/hydrus/

Experienced user with a bit of cash who wants to help out? ---> Patreon

Current to-do list has: 1,579 items

Python 3 update just happened! Check v335 release post for update instructions!

Current big job: Client API

9df3a5  No.11676[Reply]

I fell ill this week and did not get enough work done to justify a release. I will make some of the time up tomorrow and then continue as normal, aiming for 341 to be out on the 27th.

Thanks everyone!

YouTube embed. Click thumbnail to play.

9ecea0  No.11606[Reply]


zip: https://github.com/hydrusnetwork/hydrus/releases/download/v340/Hydrus.Network.340.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v340/Hydrus.Network.340.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v340/Hydrus.Network.340.-.OS.X.-.App.dmg


tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v340/Hydrus.Network.340.-.Linux.-.Executable.tar.gz


tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v340.tar.gz

I had a great if busy week. The Client API does more, several bugs are fixed, some new features and options are in, and stability and memory use should be a bit better.

client api

This is still somewhat advanced, but users who are interested may want to start looking in.

The first version of the Client API last week went ok! There were a couple of bugs, but thanks to feedback from some advanced users, I've improved reliability and carved out a better spec. This week brings JSON everywhere, fixes the add_file crash, and adds two neat calls:

/add_urls/get_url_files nowPost too long. Click here to view the full text.

21 posts and 16 image replies omitted. Click reply to view.

a4c48c  No.11669





bit late in saying it but thanks

File: 3f8a8d22364b1de⋯.jpg (111.13 KB, 400x400, 1:1, 1394958218797.jpg)

d5679d  No.9327[Reply]

Here you can ask questions so that the board is not clogged with small threads.

>>6021 has reached its bump limit, so I made a new thread.

531 posts and 104 image replies omitted. Click reply to view.

53e7e3  No.11672

File: 90ab535c271478f⋯.png (67.41 KB, 230x234, 115:117, 90ab535c271478fb9f387bb520….png)

I build a parser that works fine when I run it through the test functions in the editor, but fails to work in any of the downloaders with a "Did not have a parser for this URL!" error. What am I doing wrong?

40496e  No.11673

File: 89a09a61102bc1f⋯.png (193.95 KB, 1920x1080, 16:9, ClipboardImage.png)

File: efe87aaf413a912⋯.png (4.53 KB, 512x110, 256:55, easy-import downloader png….png)

File: ff20349c0061664⋯.png (233.74 KB, 868x680, 217:170, 3.png)

File: 12b288a6219ea42⋯.png (485.9 KB, 1250x718, 625:359, 4.png)

I'm trying to make a downloader for a shitty site with good images, but the site uses hashes for gallery groups, like "solo" would be completely different from "#solo", and that's a bit of a problem with hydrus.

If you use %23, hydrus automatically converts it to a hash, so the url goes from "q=%23solo" to "q=#solo" which the shitty site doesn't support.

I tried using regex, which should work, but something with hydrus makes the whole string after the hash disappear for some reason, which completely fucks it up.

First pic is what happens, second pic is the downloader.


Not much, for the most part. Unless you stumble upon a top kike site that uses scripts out of the ass to protect the pictures, it'll just be pattern matching.

The site that I'm doing is one of those simple ones.

I used page inspection to select the image, there I saw that all images were in a "img" container with a "class" "image_perma_img", with the picture being refered to by the "scr" thing. Basically you just tell Hydrus to look through all <img> with class=image_perma_img and get the url in the "scr".

As for the tags, every tag, whether posted by users or the OP, are <a> classes with class=hash_mentions with the text in between, so you just tell hydrus to look for that, then to remove the first character, since the site uses # before each tag.

What I usually do is to duplicate gallery/file regex then work over them, to facilitate the process.

40496e  No.11674

File: b5bde22e16a2c2b⋯.png (16.49 KB, 823x274, 823:274, ClipboardImage.png)


You need to connect the downloader to the parser.

Pic related works unless the example urls are different.

d5679d  No.11675


Decompression bombs are images that eat up all your ram while trying to render. Ke-Ta images from gelbooru are notorious for this.


Well this is helpful but I'm a complete newfag at this. I had previously tried to make a parser for ddlc booru and tried using an existing site parser as a base because the searchpatterning was the same but it didn't work out. Makes me wish there was a complex guide for it because once one guy who knows what he's doing does a write up then any newfag can make a parser for sites with relative ease.

2a5488  No.11677

I think my client's trying to sync with the tag repository, but it's throwing an exception at me, any ideas?


Network job death time reached–not sure what the error was. Maybe a paused service?

Traceback (most recent call last):

File "include\ClientGUIPanels.py", line 846, in do_it

self._service.SyncAccount( force = True )

File "include\ClientServices.py", line 993, in SyncAccount

response = self.Request( HC.GET, 'account' )

File "include\ClientServices.py", line 876, in Request


File "include\ClientNetworkingJobs.py", line 1113, in WaitUntilDone

raise Exception( 'Network job death time reached–not sure what the error was. Maybe a paused service?' )

Exception: Network job death time reached–not sure what the error was. Maybe a paused service?

File: 7a56ef4350b0bb9⋯.jpg (64.84 KB, 445x488, 445:488, 7a56ef4350b0bb96c5b6a4eb80….jpg)

ff2c17  No.11542[Reply]


24 posts and 11 image replies omitted. Click reply to view.

ff2c17  No.11652


Thank you. I will revisit this code this week. I am not sure what is happening here, but I think it is OS- or window manager-specific–is this on Linux, as running from source?

ff2c17  No.11653



Hey, I think >>11633 has it. Try with brackets, [ and ], and not parentheses, ( and ). With parentheses, it is looking for 'page' rather than the x in 'page:x', and since the files do not have 'page', the resultant filename is the empty string. Each file gets the same empty string, so my 'duplicate-filename de-deduper' is putting ' (n+1)' on the end of each one.

ff2c17  No.11654


Thank you for this report. I do not recommend you try to load that many files at once! I recommend you set a limit under options->speed and memory. 10k is the default.

That said, this bad number may be due to some file miscounting in the past. Please try running help->debug->data actions->clear db service info cache and then restart the client (you might like to note down your exact numbers here beforehand). Afterwards, have the inbox/archive/everything numbers changed? What about the 'all local files' and 'my files' file domains under services->review services panel? 'all local files' should have the same number of files as the sum of 'my files', 'repository updates', and 'trash' domains.

ff2c17  No.11655


I have had a lot of trouble getting those floating windows to work right in non-Windows, so by default I have them 'embed', like in the manage tags dialog. Please check the 'always embed autocomplete …' option under options->gui and restart the client–does that work better?

There is an ugly BUGFIX option on the same page to make the hover windows on the media viewer always show. It isn't nice, but it works better on the Linux window managers that do not like my code.

Did you by any chance migrate from Windows?

93caef  No.11667


not running from source, i am running the the executable. To be more specific, The problems happens if i rover the mouse over the pending menu after i put one to sync. I am runing ubuntu 18.04 with KDE plasma, if i go over GNOME the problem is worst, most of the menus get greyed out

a0930d  No.11455[Reply]

Does anyone have a working gallery importer for AGN.PH? Gotta collect it all…

b081a4  No.11464


Ask for a parser

cf0b1a  No.11470

I was able to script about 70% of it but ran into an issue. The site has one of those fucking click through screens. other than that it is practically identical to e621. Just with worse tagging.

000000  No.11521


>The site has one of those fucking click through screens.

You mean, the confirm age one?

That may be solved by connecting to the "yes, I am 18" URL directly.

a0930d  No.11656


it blocks that. you need A verified cookie.

File: 1426721772716.png (100.78 KB, 1624x1081, 1624:1081, 1327614072601.png)

7f2c0e  No.471[Reply]

Drag and drop windows with tag rules. Show two windows side by side and one window can be programmed with the rule "ADD tag foo" and the other one has the rule "REMOVE tag foo, ADD tag bar" and you can drag and drop files to them.

Deriving tags from regex of other tags/namespace tags. A file has the tag "filename:big_ugly_name" and we could regex that namespace for another tag.

Tag sets with hotkeys: save a set of tags under a hotkey so it's quick to add them to a file while filtering

Opaque window behind tag list in the corner so it doesn't get hidden by picture background

Option to default certain mime types to be excluded from slideshow and only open externally, will help with videos with odd codecs that don't preview in the slideshow correctly

Option to specify hamming distance in "find similar images", you can't change the option once it's in the filter window and you have to enter the hash manually in the "system:similar to" option
630 posts and 191 image replies omitted. Click reply to view.

2bf658  No.11618


but then how do we link the file's SHA256 hash to the tags if we do the former? The latter is much more doable in my opinion. So we could basically create a "folder husk" for all the shortcuts when transferring the files into one single "pool". Extra points for two shortcuts linking to one file and allows for de-duplication.

2bf658  No.11619


Moebooru Android has renamed itself as Flexbooru

d20b68  No.11625


>but then how do we link the file's SHA256 hash to the tags if we do the former?

Isn't the hashing process only done once and then the file is referred by filename afterwards? In that case it would be possible to use the shortcut to refer to the original file when opening it or doing the initial hash.

Though I do agree it would be a nightmare with file hashes not being the same filename hash, fucking with integrity checking, fucktards deleting/moving the original file and fucking with the shortcut and so on, so the latter is more doable.

bce4ba  No.11637

File error checking with PAR2 or other PArchives


48a4a8  No.11642

File: d7e685360192f2a⋯.png (20.52 KB, 722x449, 722:449, api-image-for-blog.png)

b45f6a  No.11626[Reply]

ITT: We propose new features that can be solved by using the API, and recommend new API commends for it

b45f6a  No.11627



File: ed3f745dbd39b5d⋯.jpg (4.66 MB, 4000x2715, 800:543, shutterstock_89245327.jpg)

12f73e  No.4475[Reply]

How about a thread for discussing/creating/sharing parsing scripts?

I made one for md5 lookup on e621.net (actually I just modified Hydrus_dev's danbooru script). Let me know if I did anything wrong with it, I'm pretty clueless… but it seems to work fine.

[32, "e621 md5", 1, ["http://e621.net/post/show", 0, 1, 1, "md5", {}, [[30, 1, ["we got sent back to main gallery page -- title test", 8, [27, 1, [[["head", {}, 0], ["title", {}, 0]], null]], [true, true, "Image List"]]], [30, 1, ["", 0, [27, 1, [[["li", {"class": "tag-type-general"}, null], ["a", {}, 1]], null]], ""]], [30, 1, ["", 0, [27, 1, [[["li", {"class": "tag-type-copyright"}, null], ["a", {}, 1]], null]], "series"]], [30, 1, ["", 0, [27, 1, [[["li", {"class": "tag-type-artist"}, null], ["a", {}, 1]], null]], "creator"]], [30, 1, ["", 0, [27, 1, [[["li", {"class": "tag-type-character"}, null], ["a", {}, 1]], null]], "character"]], [30, 1, ["", 0, [27, 1, [[["li", {"class": "tag-type-species"}, null], ["a", {}, 1]], null]], "species"]], [30, 1, ["we got sent back to main gallery page -- page links exist", 8, [27, 1, [[["div", {}, null]], "class"]], [true, true, "pagination"]]]]]]

41 posts and 13 image replies omitted. Click reply to view.

cc9eb8  No.9280


Hitomi, Tsumino, Hentai2Read , HentaiCafe, NHentai, HBrowse and Goddess are what /a/ recommends when avoiding SadPanda

d54ab2  No.11124

File: 75e5aeb99022928⋯.png (5.8 KB, 512x125, 512:125, e621 pool lookup.png)

Here's an e621 pool lookup.

Seems to work for me, images appear in correct order in the browser pane. I just need to find a better way of tagging page:* and title:*

atm I drag the files onto Krename which outputs to /tmp/hydrus/<title>/<page>.<ext> and use the tag based on file name import option.

9d5f02  No.11590

File: fd3e236eaa08774⋯.png (4.69 KB, 512x111, 512:111, a.png)

Just made a realbooru one.

Fucking parsers are a pain in the ass.

9d5f02  No.11610

File: ec56d235964857e⋯.png (4.98 KB, 512x111, 512:111, easy-import downloader png….png)


Wait, I fucked up. Here's the fixed version.

354bf6  No.11616


Can any custom parsers handle logins? Like the twitter gallery situation is still out of the picture and has been for a few months now. Fur Affinity and InkBunny if parsers are made but without logins will barely scrape any content as well. I know Hdev said FA gallery parser is coming but without login support it's hardly worth the work to make one imo.

YouTube embed. Click thumbnail to play.

e21f89  No.11548[Reply]


zip: https://github.com/hydrusnetwork/hydrus/releases/download/v339/Hydrus.Network.339.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v339/Hydrus.Network.339.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v339/Hydrus.Network.339.-.OS.X.-.App.dmg


tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v339/Hydrus.Network.339.-.Linux.-.Executable.tar.gz


tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v339.tar.gz

I had a good week. I fixed some bugs and finished one part of the Client API.

client api

(This is only for advanced users for now.)

The Client API is now ready for some experimentation! Although most of this work has gone much faster than I expected, the last step of actually writing the server commands along with some half-decent test code and help documentation is proving a little slower. This week rolls out 'set up API access' and 'add urls' functionality. In the coming weeks, I will roll out 'add file', 'add tags', and 'search files' commands.

The basic (under coPost too long. Click here to view the full text.

22 posts and 5 image replies omitted. Click reply to view.

e21f89  No.11598


After a bit more poking around, I think regular garbage collection wasn't running for some reason. I'll do some more work on this today, it may be related to another issue.

e21f89  No.11599


Thank you for this report. Just to check, this is non-Windows, right? Did you happen to migrate an install from Windows? I have had the worst luck trying to get the floating version of the autocomplete to work on non-Windows for these sorts of focus issues, so Linux and OS X installs start with it always in embedded mode, like how the 'manage tags dialog' works.

Please hit options->gui->always embed autocomplete dropdown results window and let me know if that works better.

7782ef  No.11603

File: ed96a686209f415⋯.png (2.38 MB, 2096x2755, 2096:2755, ed96a686209f415c58fcb98791….png)


Thank you, it's working now. I almost was about to post that it still wasn't working but the client crashed when I was about to take a screenshot and it started to work after the restart.

7782ef  No.11604


Oh, and to add to your question: I indeed have migrated from Windows long time ago. I just tried a fresh install and it defaults to that setting, so the problem was just outdated defaults. If I wasn't pantsu-on-head retarded, I would've checked if it was working on a fresh install first. Sorry for wasting time.

740a49  No.11605


I reinstalled 339 and havent noticed the memory spike as bad as before but it is locking up and stops responding pretty often so i still end up having to restart anyways for any downloading. the program still works for main archival and search though.

File: 47aff7a65fdde59⋯.gif (244.75 KB, 774x900, 43:50, 47aff7a65fdde59c7f5569108a….gif)

15986e  No.11601[Reply]

I had a great if busy week. I fixed the stupid page of pages bug (and a whole bunch more), fleshed out the new client api, added some small new options, improved stability, and added several new downloaders to the client defaults.

The release should be as normal tomorrow, maybe a little late.

File: ea31ab670c8dee7⋯.png (12.55 KB, 340x175, 68:35, mariadb-usa-inc.png)

928501  No.9068[Reply]

ITT: create proposals for making Hydrus more optimized.

Proposal: Why can't Hydrus switch to MariaDB?

If it is faster, then it should be better. The only trouble is having the need to rewrite the queries, which from an SQL standpoint should be a non-issue, right?

List of Databases with Open Source License and Open Source APIs:

SQLite - Currently used in Hydrus, has minimal features

MySQL - A more well-rounded SQL Database with user management

PostgreSQL - An SQL with complex features with less performance

MariaDB - SQL/NoSQL database with heavy optimizations

ElasticSearch - A literal search engine instead of a normal Database

Teradata - IDK



45 posts and 2 image replies omitted. Click reply to view.

74b0c2  No.11133


>PostgreSQL - An SQL with complex features with less performance

1998 wants it retard memes back.

ed0a1d  No.11204

File: 880feed1fc57634⋯.png (1.57 KB, 300x300, 1:1, 下.png)



928501  No.11206


How so? Too many onyomi and kunyomi? Even then if we are not using phonetic fuzzy search, string fuzzy search can still be used (see https://en.wikipedia.org/wiki/String_metric)

71a1b2  No.11380

138bc8  No.11586

0bdeb8  No.11529[Reply]

need the size of the thumbnail more than 200x200.

on 4k TV, they look small

59ba7c  No.11546

Read >>10203 then write up a proper proposal.

File: c1adee5f5ad8a8b⋯.gif (710.97 KB, 600x400, 3:2, c1adee5f5ad8a8bc16c3f778d7….gif)

d843d0  No.11544[Reply]

I had a good week. I fixed a lot of bugs (including a couple of stupid ones I let slip into 338) and finished a first subset of the client api for advanced users to play with.

The release should be as normal tomorrow.

Post last edited at

File: ab23431121b8563⋯.jpg (96.69 KB, 492x650, 246:325, ab23431121b8563a7bc3b7ac57….jpg)

0675b5  No.5115[Reply]

Gonna start another thread because >>173 is becoming too big for most people.

In the tag manager, whenever multiple tags are selected for multiple files with different tags, if you press the del key, the program will still ask you if you want to delete them all or put them all in the selected files. This does not make sense, if I'm pressing the del key it's obvious that I want them gone.

643 posts and 201 image replies omitted. Click reply to view.

35b3df  No.11526


Thank you, this vid is very helpful. It looks like you have a pretty heavy client, with a lot of pages open and a ton of pending tags. Do you have, say, more than 100 pages open? I am wondering if your client is running out of ids for menus quicker because it has all the other stuff going on. Is it possible you can temporarily close some of those pages, restart the client, and see if that eases things up a bit?

Have you tried 'page of pages', btw? This helps nest pages Hit F9 (i.e. choose a new page) and then special->page of pages. You can drag and drop page tabs onto that new page 'folder' and you won't be navigating left and right across a long ribbon of tabs.

35b3df  No.11527



Thank you for this report. I apologise, I fucked up a change here. I will have it fixed for 339.

533e30  No.11532

Hi, having a small problem with the Manage Import Folders option.

CallBlockingToWX() missing 1 required positional argument: 'func'
Traceback (most recent call last):
File "include\HydrusThreading.py", line 342, in run
callable( *args, **kwargs )
File "include\ClientGUI.py", line 2608, in THREAD_do_it
controller.CallBlockingToWX( wx_do_it )
TypeError: CallBlockingToWX() missing 1 required positional argument: 'func'

Not a problem with export folders.

35b3df  No.11535


Thank you for this report. I made a couple of stupid mistakes last week–this was one. I am sorry for the problem. It will be fixed in tomorrow's release.

35b3df  No.11543

This thread is getting laggy, so:















File: fa486f8bbe5b89a⋯.mp4 (11.61 MB, 1280x720, 16:9, yata.mp4)

2eb52a  No.11493[Reply]


zip: https://github.com/hydrusnetwork/hydrus/releases/download/v338/Hydrus.Network.338.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v338/Hydrus.Network.338.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v338/Hydrus.Network.338.-.OS.X.-.App.dmg


tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v338/Hydrus.Network.338.-.Linux.-.Executable.tar.gz


tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v338.tar.gz

The recent v335 was an important update. If you are updating from v334 or earlier, please go read that release post for your special one-time update instructions and update to v335 before you try this week. Once you have updated to v335, the process for updating to anything newer is back to normal.

I had a great week. There's some bug fixes and neat new tag stuff.


After talking to some users this week, I decided to knuckle down and improve autocomplete tag input. This has been in the works for a long time, but I have had trouble just putting the time in.

Post too long. Click here to view the full text.
20 posts and 4 image replies omitted. Click reply to view.
Post last edited at

7b3f15  No.11534


windows offline updater, and look for a list of suspect updates and hard block them, that way you have all the updates with none of the bullshit. and this is a good 2400% faster then normal windows update if I remember correctly.

2eb52a  No.11536


Thank you for this report. This is odd and interesting. My first guess is perhaps that file got reparsed at some point with newer code/ffmpeg, so while the db entry considers it more accurately now an mkv, the file structure does not.

Do you ever do right-click->repare and regen thumbs on your movies?

I had not considered this when I wrote this routine (I was thinking of frame counts and bad thumbs at the time). I will update the routine so it updates the file storage as well, and I'll see if I can get it to auto-heal retroactive extension problems like this too. I may not have time to do this today, but please let me know how you get on with this as I roll it out.

If you would like to fix it immediately, you can just manually rename the file's extension in client_files. Afaik, that's the only problem here.

If you believe the file really should be detected as a webm rather than mkv, and it isn't private, please post it here or email it to me and I'll have another look at my parsing.

2eb52a  No.11537


Thanks, I will.

d92b37  No.11540


>Do you ever do right-click->repare and regen thumbs on your movies?

Yes, in fact I did this on all my videos because I'm the guy who had Hydrus randomly generate broken green thumbnails for videos (which seems to be fixed now by the way, if not by you then perhaps because I got a new computer and switched from Windows 7 to 10).

An auto-fix would be nice because it has happened to a lot more than just that one video.

Is webm and mkv very close in how the files are structured? Because most of these were downloaded from gelbooru which lists them as webm. But I guess it doesn't really matter if the hash stays the same.

2eb52a  No.11541


Yeah, iirc, a webm is just an mkv with VP8/9 video and vorbis audio, I think. All webms are mkvs–they are a subset. I think the idea was to guarantee a limited mkv format that would work on all devices for HTML5 streaming etc…

I could also have some of my ffmpeg parsing wrong here, or had it wrong when you did this reparse. I used to use an external library to figure this out, and now I compare ffmpeg output specifically. If you run the reparse again, maybe it'll flip back to webm!

Anyway, should be fixed for tomorrow. Let me know if you still have any trouble.

Delete Post [ ]
Previous [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15]
| Catalog | Nerve Center | Cancer
[ / / / / / / / / / / / / / ] [ dir / abdl / animu / doomer / leftyb / lovelive / mde / tingles / zoo ]