[ / / / / / / / / / / / / / ] [ dir / b2 / b4 / choroy / coz / fanfic / fascist / marxism / radfem ]

/hydrus/ - Hydrus Network

Bug reports, feature requests, and other discussion for the hydrus network.

Catalog   Archive

Winner of the 83rd Attention-Hungry Games
/strek/ - Remove Hasperat

May 2019 - 8chan Transparency Report
Name
Email
Subject
Comment *
File
Password (Randomized for file and post deletion; you may also set your own.)
* = required field[▶ Show post options & limits]
Confused? See the FAQ.
Embed
(replaces files and can be used instead)
Options

Allowed file types:jpg, jpeg, gif, png, webm, mp4, swf, pdf
Max filesize is 16 MB.
Max image dimensions are 15000 x 15000.
You may upload 5 per post.


New user? Start here ---> http://hydrusnetwork.github.io/hydrus/

Experienced user with a bit of cash who wants to help out? ---> Patreon

Current to-do list has: 1,793 items

Current big job: Finishing duplicate db overhaul and filter workflow improvements


YouTube embed. Click thumbnail to play.

a3edbc  No.13223[Reply]

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v360/Hydrus.Network.360.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v360/Hydrus.Network.360.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v360/Hydrus.Network.360.-.OS.X.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v360/Hydrus.Network.360.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v360.tar.gz

I had a great week. There are a bunch of little things and an important speed overhaul to tag autocomplete.

tag autocomplete

I have never really liked the tag autocomplete workflow. It once blocked the UI completely, and some unusual timing options were needed to make it even useable, but it still often lagged out or just responded with gigantic lists in judders. After chipping away at the problem, this week I am finally updating it to what I really wanted.

So, the main change is that autocomplete results are now fetched as soon as you type. It responds very quickly and overall, I think, feels great, particularly once you have put fourPost too long. Click here to view the full text.

3 posts and 1 image reply omitted. Click reply to view.

8897f1  No.13229

Wow wow wow tagging feels so much snappier now. Loving it! Thank you so much!




File: 7a56ef4350b0bb9⋯.jpg (64.84 KB, 445x488, 445:488, 7a56ef4350b0bb96c5b6a4eb80….jpg)

ff2c17  No.11542[Reply]

BUGS THREAD

217 posts and 64 image replies omitted. Click reply to view.

c6e696  No.13208

>>13198

No worries. Everyone's busy and there's no rush.

What makes me doubt the graphics driver is the import working fine with a fresh database.

Could there be anything else with my database holding back the import?

Thousands of other files (yes, I am a hoarder) have imported fine since that particular one. So generally everything works.


26d44e  No.13213

>>13204

Thank you. The files do not get a potential pair for me, so this is not some legit false positive accident. There is definitely something going wrong in your db. I recommend you pause automatic idle time potential pair searching by clicking on the cog icon on the duplicates page. You'll probably want to completely reset your potential duplicates as well through the same menu, once we have figured this out.

Can you please close your client and go to install_dir/db and run the sqlite3 executable to bring up the sqlite terminal, and then copy/paste these lines in one by one:

.open client.db
ATTACH "client.master.db" as cm;
ATTACH "client.caches.db" as cc;

SELECT phash_id, hex( phash ) FROM hashes NATURAL JOIN shape_perceptual_hash_map NATURAL JOIN shape_perceptual_hashes WHERE hash IN ( x'232d97e6126a25b8ec4ae36f65b5e8ae08314e74d06f4fcbc72350a9fd3f5995', x'6ed82640fdd14643660cb4f50f022b5b0dd7a4e1c4ff3f6d430a5ed8e7c82fac' );

SELECT media_id, hash_id FROM duplicate_files, hashes ON (king_hash_id = hash_id ) WHERE hash IN ( x'232d97e6126a25b8ec4ae36f65b5e8ae08314e74d06f4fcbc72350a9fd3f5995', x'6ed82640fdd14643660cb4f50f022b5b0dd7a4e1c4ff3f6d430a5ed8e7c82fac' );

SELECT media_id FROM duplicate_file_members NATURAL JOIN hashes WHERE hash IN ( x'232d97e6126a25b8ec4ae36f65b5e8ae08314e74d06f4fcbc72350a9fd3f5995', x'6ed82640fdd14643660cb4f50f022b5b0dd7a4e1c4ff3f6d430a5ed8e7c82fac' );

(note those media_ids down, and put them in here a couple times):

SELECT * FROM duplicate_file_members WHERE media_id = (media_id here);

.exit

Does that first SELECT statement give you:

(some number)|EAE648A595946D8B

(some other number)|E8EAD2E49595C583

Or does it give very different numbers, or a much bigger list?

Does the duplicPost too long. Click here to view the full text.


26d44e  No.13214

>>13205

Hmm, I am not sure what it doesn't like there. I don't know enough about subprocess details. I have not had luck with anything but subprocess.Popen. If you do something like this:

p = subprocess.Popen( ['exiftool.exe', argv[1]], stdin = subprocess.PIPE, stdout = subprocess.PIPE, stderr = subprocess.PIPE, universal_newlines = True, encoding = 'utf-8' )

( stdout, stderr ) = p.communicate()

That may work out better, I am not sure. Maybe doing the PIPE stuff is ok, since I am already doing that to call your process and there is some sys-related std-'chaining' issue going on? I dunno though, because it looks like the .run command just calls Popen anyway deeper into subprocess.

Was that traceback you got hydrus-side written to the client.log in install_dir/db? Does it show the full trace there, so we can see the actual meat of what it was having trouble with?


26d44e  No.13215

>>13208

Shit, I forgot that it was ok in a fresh db. Unless something very odd is going on that is prohibiting the next message getting out or being flushed to the log or something, after that 'generating phashes' line should be 'File import job generated xxx phashes'. Since that message is not coming up, it suggests the hang is inside the pure-CPU graphics processing code that generates similar files phashes. This all happens before the db is opened for the actual file import.

Ok, let's see how this new phash generating report mode goes in tomorrow's release.


dcc7c5  No.13227

I tried to import an epub but hydrus somehow turned it into a zip file.

This is the file

https://files.catbox.moe/d6005p.epub




File: b28babaa13b7dbf⋯.png (765.02 KB, 1278x720, 71:40, b28babaa13b7dbfb92d271ee0a….png)

136209  No.12641[Reply]

Here you can ask questions so that the board is not clogged with small threads.

Old thread >>9327

205 posts and 29 image replies omitted. Click reply to view.

93df13  No.13217

>>13211

Yes, that makes a lot of sense. I agree about stoicism. I happened upon your recommendation of Marcus Aurelius to another guy, and it's good stuff. Some self-discipline is certainly in order, which cannot be provided by software.

I certainly intend to give some feedback after I've had a chance to organise my thoughts. I'm sure most of it, if not all, will have been said already though so hopefully it's not too tiresome reading duplicate suggestions. Can't really see a public to-do list (or the like) around to help prevent that, though.


11928b  No.13220

>>13192

Updating to the latest version seems to have solved the issue. When i started the client i got the prompt that the db entries were missing, and i could then point them to the folder where my files are stored. Will do some integrity checks now to see if anything else breaks.

Thanks for the help!


86f6c5  No.13221

File: be3abb69baee671⋯.png (330.79 KB, 594x640, 297:320, 1557349478670.png)

Is there a way to force-download the whole tag set of the PTR? I increased the max bandwidth for Hydrus services but it's saying

>service has exceeded bandwidth - next request in 5 days 17 hours

I know there was a way to do it in a previous release but I can't find it now.


86f6c5  No.13222

>>13221

Ah, never mind. I found the option to import.


7f1898  No.13226

File: 251535f27a89464⋯.png (948.46 KB, 791x725, 791:725, 251535f27a894644416f8e13c6….png)

I already have the PTR downloaded, is there any way I can export it and import it on a separated hydrus?




File: 4d6f470068e6acc⋯.jpg (338.2 KB, 1925x1080, 385:216, 1517368870.jpg)

da8629  No.12692[Reply]

To quit spamming the Q&A thread (sorry) I agree this should be made into its own thread.

I'd like this to be a place to ask questions on "how should I tag this" for those that don't use the PTR and think you may be able to be more intelligent with how you tag. Also a place to discuss how you tag things and, most important, WHY that way. For example, what sort of issue were you looking to solve or what did it make easier by doing it that way?

T

o start off the discussion - I'm going to write a small essay on my thoughts on tags/tagging. It is my belief that tagging should be, above all else, two things

1) Simple - There are few scenarios where you must think if something should be tagged or not. Edge cases and oddities should be minimized at every opportunity. The number of tags should be kept small enough to maintain the entire list in memory. This is important for consistency. Speaking of which, the second item is…

2) Consistent - Tags should be well defined and used consistently. If a tag is often forgotten to be applied then it is a bad tag and should be either removed from the system or renamed/redefined as to be applied more consistently. In order to keep consistent tags it is also a requirement that all tags are able to be kept in memory to be applied properly. If you have so many tags that you are incapable of remembering them all then you will likely be tagging inconsistently. It is okay to add new tags to cover areas.

A tag that is used infrequently because the number of items in the gallery are too few is acceptable but is discouraged if the term is rarely searched for (either as an inclusive or exclusive term). Remember: tags are meant to help you search for images. This is important when it comes to naming tags.

I use a collection of 288 unnamespaced tags and 11 namespaces. The tags are mostly descriptors of clothes, physical traits, and objects. Namespaces are used for more meta information like character: artist: series: or whether a piece of art is SFW or pornographic.

Some of the more heaPost too long. Click here to view the full text.

7 posts and 3 image replies omitted. Click reply to view.

0a9d65  No.12741

> Metholodogy

Why not we gather all the major ways of managing Hydrus and create a "common workflow"?


9d3e84  No.12748

File: 76ab2e2ef12fa4e⋯.swf (8.56 MB, 76ab2e2ef12fa4e51dbe4d865a….swf)

I'm not sure if this goes here or in the questions thread, but how should I tag flash codes in the PRT?

Boogie usually puts scenes in his flashes that are only accessible by code. Flash related has two alternate scenes you can access by typing 'paper' and 'yokai'. I think those are useful enough that I should put them in the PTR, but I don't think anyone has done anything similar, so I'm not sure what namespace to use. Maybe 'code:' or 'cheat'?


c87514  No.13156

>>12741

This is a good idea.


98b17a  No.13210

>>12748

That's not really something that should go into the tags at all, it's not really something you search with. If the file's notes could be shared on the public server it would go in there


0a9d65  No.13219

>>13156

Wanna pitch in on how things can be done, judging by this thread? Of course the more variations the merrier




File: f2bae29e9c77713⋯.gif (102.71 KB, 220x166, 110:83, f2bae29e9c77713a2456968a02….gif)

9dbd5d  No.13218[Reply]

I had a great week. As well as fixing bugs and catching up on small jobs, I put some time into the new file maintenance manager and significantly reworked and improved the search timings of the tag autocomplete system. Tag results now come in as fast as they can–it feels completely different.

The release should be as normal tomorrow.



YouTube embed. Click thumbnail to play.

ae2c5f  No.13160[Reply]

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v359/Hydrus.Network.359.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v359/Hydrus.Network.359.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v359/Hydrus.Network.359.-.OS.X.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v359/Hydrus.Network.359.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v359.tar.gz

I had an excellent and very full week.

file maintenance

The new file maintenance system now has some nice UI under database->maintenance->review scheduled file maintenance. It has two pages. It is a little advanced, but if you have some hydrus experience, please feel free to have a play around with it.

The first page reviews what is currently scheduled to run. It shows job counts for each type and lets you manually start work on that job type or cancel them.

The second lets you schedule new jobs. It uses the standard file search interface, so you can queue up aPost too long. Click here to view the full text.

7 posts omitted. Click reply to view.

e05f8e  No.13181

>>13175

just looked at the old version and saw the reply

for multiple similar images at once, the only real use I can think of it is I have a full comic/set and I want to remove the images, I have images that hit an 'unappealing' tag, and I want to get every one of them at once, or in my case, I want that tails comic wiped from wasting space. at least for me, haveing them sort is kind of a moot point.

however if you want them to sort, I think adding a queue system to duplicate filter and having files open in a duplicate filter window would work. this way you search one files duplicates, it posts results and goes onto the next files results and duplicates. this would allow you to also have some access to duplicate tools.

as for the other one, I don't know how simple it is, but the database knows that the files were thrown away, so a 'remove deleted from view' would probably do the job, unless you want to expand the 'remove from view' options to be a bit more complex and have their own sub menu.

It could also be handled from a highlight all function, this would add another step to removing deleted from view, but it would also allow something like 'highlight all 'jpeg'' so I could move them to a different window. it may not have a whole lot of use case for search results, but in the case of publishing a subscription, I think it would have a fair amount of use.


03e697  No.13187

I use hydrus on an external drive and the other day my drive came unplugged accidentally while using hydrus and the program closed as you'd expect it would. Though when I reconnect the drive and try to reopen hydrus I get to "initializing managers tag siblings" then hydrus crashes. Any advice? I have a backup but its a couple weeks old and I'd prefer not to use it. I still have my db folder and client file and everything. Here's the command I use to launch hydrus just for reference. I'm on osx btw.

open -a /Applications/Hydrus\ Network.app –args -d=/Volumes/Drive/hydurs/db


ae2c5f  No.13188

>>13178

I am glad you like the sound of it. The program does not phone home anywhere, but if you are thinking about using the program and are truly skeptical, I recommend you block it with your firewall to start with. It shouldn't try to make any connection unless you tell it to through a booru downloader etc…

If you aren't a user yet, your enthusiasm may dim a bit when you stumble over some of the unprofessional design or buggy code, though! Everything is breaking all the time.

I am a typical Anon and am trying to make the sort of software I want to use. Privacy and user control is very important to me, so it is baked into the program. I love the imageboard community and enjoy giving back to it through my work. If you do use the program, let me know how you overall find it after a few weeks, once you are comfortable with it. Feedback from new users is always useful.


ae2c5f  No.13189

>>13179

Great, thanks for letting me know. I'm not anywhere near ready to start work on this, but next step for the whole downloader system will be some sort of subscription + version checking system for the various scripts, like you'd see in Nexus Mod Manager, so script authors can update people more easily when sites change like this.


ae2c5f  No.13190

>>13181

Thanks, I will keep thinking about this.

>>13187

Hmm, that is odd. When it crashes, does it throw up any error window at all, or just close itself? Does the close look 'clean', or does it just die in one frame?

My first guess is the disconnection caught the db while a transaction was flushing to disk and so there is some bad data in there, although usually that stuff is recovered from.

If you go to /Volumes/Drive/hydurs/db and check your latest client.log file and scroll to the bottom, is there any traceback error there for what happens after the 'booting db' phrase? A db error generally shouldn't cause a crash, so it is also odd that you aren't getting an UI popup of some sort saying 'hey, the db had a problem on boot'. If it is erroring silently and quitting the program neatly, there could be some error in there, but if it is a full-on actual crash, there won't be any kind of tidy-up statements.

If there is no error, let's check your db files. Here's a help file that is a slight pain in the neck to get to as an OS X user:

https://github.com/hydrusnetwork/hydrus/blob/master/db/help%20my%20db%20is%20broke.txt

It will walk you through some integrity checks to make sure your db is all ok. I think there is an OS X sqlite3 executable in your App, if you know how to navigate that, or you can get the tools here:

https://sqlite.org/2019/sqlite-tools-osx-x86-3290000.zip

Let me know what you find. I am somewhat confident we can recover this db, but we might need to reset somethinPost too long. Click here to view the full text.




YouTube embed. Click thumbnail to play.

9bcbc0  No.13106[Reply]

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v358/Hydrus.Network.358.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v358/Hydrus.Network.358.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v358/Hydrus.Network.358.-.OS.X.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v358/Hydrus.Network.358.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v358.tar.gz

I had a great week doing duplicates work and fixing bugs.

duplicates

I split this big duplicates storage overhaul into three jobs, and this week marks the third and final job done. Like alternates and duplicates information, potential pairs are now stored in a unified and more efficient way.

On the front end, you may notice your potential pairs queue shorten again on week. It will also shrink faster as you process in the duplicate filter, which will present more 'useful' duplicate pairs first and apply your decisions more intelligently at the db level.

All the code is simplerPost too long. Click here to view the full text.

10 posts and 2 image replies omitted. Click reply to view.

fda833  No.13149

Thank you based dev.


89520c  No.13152

Is there any way to search for specific duplicate types?

I like to replace files with the best version available, but I fucked up and left a lot of "this is worse" versions around, and I'm not sure if I'm retarded or new versions prevented generic ">0 this is worse" searches.


aa6065  No.13175

>>13144

Searching for multiple 'similar files' from right-click is probably doable. Sorting them the other end is probably a hassle, but we'll leave that for later. I'm supposed to be extending system:hash to take multiples soon, so I can probably do them together.

>>13145

That's a good idea. I would say hit right-click->select->(something) where 'something' would be like 'remote' files or similar and then hit remove in a second action, but I can't remember if there is a good option to select those files. I'll make sure that select menu is comprehensive and see if I can do a quick submenu that that does remove. Maybe if no files are selected, the remove menu instead branches to a similar select-type submenu. I often do 'select->archived … remove', so it would be nice to do that kind of stuff in one action.


aa6065  No.13176

File: 488be98eb47f43a⋯.png (4.34 MB, 1819x1786, 1819:1786, 488be98eb47f43a77a355bdd00….png)

Post last edited at

aa6065  No.13177

>>13152

Not precisely. I don't have the help written yet to explain this nicely, but duplicate files are now stored in a valueless group that has a single best 'king' file.

In the coming weeks, I will add something like 'system:is_king'. I think if you combine something like [ 'system:num_duplicates: duplicates>0', 'system:is_not_king' ], that will let you search these non-king (i.e. worse) files while excluding the good ones, and you can then deleted them en masse.

Now I have this simpler duplicates tech, I expect it to slowly appear in more locations. A nice plan is to have the main media viewer say 'hey, this file has a worse quality dupe', a little like how danbooru shows this info in a file page, and have a quick way to flick to that without interrupting the main 'list' of files the media viewer is looking at. See if the new search predicates help you do what you want here, and let me know what else would work for you.




File: 1426721772716.png (100.78 KB, 1624x1081, 1624:1081, 1327614072601.png)

7f2c0e  No.471[Reply]


Drag and drop windows with tag rules. Show two windows side by side and one window can be programmed with the rule "ADD tag foo" and the other one has the rule "REMOVE tag foo, ADD tag bar" and you can drag and drop files to them.

Deriving tags from regex of other tags/namespace tags. A file has the tag "filename:big_ugly_name" and we could regex that namespace for another tag.

Tag sets with hotkeys: save a set of tags under a hotkey so it's quick to add them to a file while filtering

Opaque window behind tag list in the corner so it doesn't get hidden by picture background

Option to default certain mime types to be excluded from slideshow and only open externally, will help with videos with odd codecs that don't preview in the slideshow correctly

Option to specify hamming distance in "find similar images", you can't change the option once it's in the filter window and you have to enter the hash manually in the "system:similar to" option
696 posts and 200 image replies omitted. Click reply to view.

3718ce  No.12920

File: a2755c52c5c2aec⋯.png (98.19 KB, 456x526, 228:263, recent-searchs.png)

recent tag search


b63e40  No.12931

More awareness of exif/xmp and other embedded metadata.

For example, dupe checker being able to tell that two jpgs are 100% identical when metadata is stripped. Maybe even automatically mark as "same quality".

I've played around a bit with exiftool,

( https://www.sno.phy.queensu.ca/~phil/exiftool/ )

And a lot of photoshop jpgs have metadata about the original name it was saved under and even ID numbers pointing to the source image if it's an edit.


000000  No.12980

Sorry for not splitting this off into this thread in the first place. Here's a copy pasta:

Would you be averse to fleshing out the hover tag pane on the left of the media viewer with more functionality? I picture a possible workflow inside the media viewer being:

1. text input gets captured automatically by a new search box added atop the hover tag pane when pane has focus.

2. The hover tag pane shows tags already on the image when the search box is blank. When search box contains text, it's populated with autocompleted tag suggestions, and any autocomplete suggested tags that are already on the image have a highlighted gold background.

3. If there's no autocomplete tag suggestions, then the tag doesn't exist and therefore the Enter key should create and append the typed tag to the image. Idk what the performance implications of this would be, so maybe namespacing can be used to cut down the tag pool so it's less laggy?

4. up/down arrows can navigate autocompleted tag suggestions without needing to use mouse input to change focus from the pane. Enter key appends tag to image, delete key removes tag from image. Shift/ctrl click to select multiple tags perhaps?

5. Left/right arrows can navigate to the next image without needing to change focus via mouse click.

6. right click>mange>file tags for more advanced tagging


634157  No.13151

wish i could have multiple pages open with 90,000+ images each without lag

reeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee


de983a  No.13164

A way to exclude or only include items in a search that I've marked as an inferior duplicate or superior duplicate




File: ed3f745dbd39b5d⋯.jpg (4.66 MB, 4000x2715, 800:543, shutterstock_89245327.jpg)

12f73e  No.4475[Reply]

How about a thread for discussing/creating/sharing parsing scripts?

I made one for md5 lookup on e621.net (actually I just modified Hydrus_dev's danbooru script). Let me know if I did anything wrong with it, I'm pretty clueless… but it seems to work fine.


[32, "e621 md5", 1, ["http://e621.net/post/show", 0, 1, 1, "md5", {}, [[30, 1, ["we got sent back to main gallery page -- title test", 8, [27, 1, [[["head", {}, 0], ["title", {}, 0]], null]], [true, true, "Image List"]]], [30, 1, ["", 0, [27, 1, [[["li", {"class": "tag-type-general"}, null], ["a", {}, 1]], null]], ""]], [30, 1, ["", 0, [27, 1, [[["li", {"class": "tag-type-copyright"}, null], ["a", {}, 1]], null]], "series"]], [30, 1, ["", 0, [27, 1, [[["li", {"class": "tag-type-artist"}, null], ["a", {}, 1]], null]], "creator"]], [30, 1, ["", 0, [27, 1, [[["li", {"class": "tag-type-character"}, null], ["a", {}, 1]], null]], "character"]], [30, 1, ["", 0, [27, 1, [[["li", {"class": "tag-type-species"}, null], ["a", {}, 1]], null]], "species"]], [30, 1, ["we got sent back to main gallery page -- page links exist", 8, [27, 1, [[["div", {}, null]], "class"]], [true, true, "pagination"]]]]]]

60 posts and 18 image replies omitted. Click reply to view.

000000  No.13054

>>13052

Hmm, I've found that the gelbooru one actually works off and on. Sometimes it oddly just returns a list with 4 crosses, instead of a list of actual tags though. Now then, what I'd really like to do is automate running file look up scripts on more than one file and automatically apply all tags to each file. There doesn't seem to be away to do this through the interface when more than one file is selected, but there has to be a way, right?


caaabf  No.13056

Hit the same pixiv issue just now. The login itself doesn't seem to be the issue, I reset and redid the login within Hydrus but that seems to have changed nothing.


d53afb  No.13057

File: 97eef4a1eb57753⋯.png (4.18 KB, 512x93, 512:93, pixiv file page api parser.png)

Pixiv changed their API so the parser had to be redone. You can replace the old one with this one or wait until Wednesday as it should be in the next release. Also pixiv added captcha to login so you have to import cookies manually now. The login in hydrus won't work.


129370  No.13138

File: 40505a535d29aff⋯.png (3.58 KB, 512x111, 512:111, newsankakuparser.png)

The sankaku parser someone posted on this board that was supposed to remove the 2000 files limit didn't work properly for me, due to the naive way the parser fetched the next gallery page data I think, so I made a fix some while ago that works on my machine (TM). Please let me know if it works on yours, too.


000000  No.13162

>>13138

Working a treat right now. I understand a bit of html, but these parsers make no sense to me. Maybe I'll sit down and spend time to figure out how to do this myself sometime.




File: c166441ade92a5e⋯.jpg (86.55 KB, 640x545, 128:109, c166441ade92a5e23d139088ca….jpg)

dfbac7  No.13153[Reply]

I had an excellent and very full week. I have extended the new file maintenance system and written some UI to manage it, added some undo/reset functions for the new duplicate system, fixed a variety of bugs, and figured out an advanced-users-only extention to the IPFS service to permit the experimental 'nocopy' pin mode,

The release may be late tomorrow, as there is more than usual to test.

b2ac94  No.13154

>>13153

> and figured out an advanced-users-only extention to the IPFS service to permit the experimental 'nocopy' pin mode

Please dumb it down to the masses such that IPFS can be used by any pleb.

HACK THE PLANET!


9905e0  No.13157

is it possible to delete whole filename: from PTR?


7aef42  No.13158

>>13154

It'd be nice if we can figure out an easy-go mode for IPFS. We'll see how this works for the advanced users, and I expect I'll figure a read-only easy mode downloader for multihashes using http gateways, and we'll iterate off that.

>>13157

Not easily yet, but once I am done with the current duplicates work, I'll be adding serverside and clientside tag filters to tag repositories so both ends can prohibit types of tags automatically.




File: 442eb99a34c2ff7⋯.jpg (120.39 KB, 379x834, 379:834, 442eb99a34c2ff7c026c477316….jpg)

e65484  No.8396[Reply]

ITT post/discuss/jerk off to/improve scripts to use with hydrus

One of my scripts was broken as shit and I just fixed, but I figured it was better to graduate from the Q&A thread and make it easier to find.

14 posts and 3 image replies omitted. Click reply to view.

e8e265  No.12602

File: 95ef85ccc17bb8d⋯.jpg (58.11 KB, 804x600, 67:50, 95ef85ccc17bb8da219b1d4081….jpg)

Here's a quick python one for md5 search.

import sys 


if(len(sys.argv) < 2):
print("script.py file.txt")
sys.exit(0)


fileName = sys.argv[1]
file = open(fileName)

gel = open(fileName+"-gelbooru.txt", 'w+')
e621 = open(fileName+"-e621.txt", 'w+')
r34xxx = open(fileName+"-rule34xxx.txt", 'w+')
dan = open(fileName+"-danbooru.txt", 'w+')
tbib = open(fileName+"-tbib.txt", 'w+')
xb = open(fileName+"-xbooru.txt", 'w+')
safe = open(fileName+"-safebooru.txt", 'w+')

for line in file:
gel.write("https://gelbooru.com/index.php?page=post&s=list&tags=md5%3a"+line)
e621.write("https://e621.net/post/index/1/md5:"+line)
dan.write("https://danbooru.donmai.us/posts?page=1&tags=md5:"+line)
r34xxx.write("https://rule34.xxx/index.php?page=post&s=list&tags=md5%3a"+line)
tbib.write("https://tbib.org/index.php?page=post&s=list&tags=md5%3a"+line)
xb.write("https://xbooru.com/index.php?page=post&s=list&tags=md5%3a"+line)
safe.write("https://safebooru.org/index.php?page=post&s=list&tags=md5%3a"+line)

Just open the files you want to search in hydrus, select all, right click, select -> copy -> hashes -> md5, wait forPost too long. Click here to view the full text.


e8e265  No.12604

>>12602

I always forget to close shit. Depending on your computer that script might fuck some files up, here's the fix.

import sys 


if(len(sys.argv) < 2):
print("script.py file.txt")
sys.exit(0)


fileName = sys.argv[1]
file = open(fileName)

gel = open(fileName+"-gelbooru.txt", 'w+')
e621 = open(fileName+"-e621.txt", 'w+')
r34xxx = open(fileName+"-rule34xxx.txt", 'w+')
dan = open(fileName+"-danbooru.txt", 'w+')
tbib = open(fileName+"-tbib.txt", 'w+')
xb = open(fileName+"-xbooru.txt", 'w+')
safe = open(fileName+"-safebooru.txt", 'w+')

for line in file:
gel.write("https://gelbooru.com/index.php?page=post&s=list&tags=md5%3a"+line)
e621.write("https://e621.net/post/index/1/md5:"+line)
dan.write("https://danbooru.donmai.us/posts?page=1&tags=md5:"+line)
r34xxx.write("https://rule34.xxx/index.php?page=post&s=list&tags=md5%3a"+line)
tbib.write("https://tbib.org/index.php?page=post&s=list&tags=md5%3a"+line)
xb.write("https://xbooru.com/index.php?page=post&s=list&tags=md5%3a"+line)
safe.write("https://safebooru.org/index.ph
Post too long. Click here to view the full text.

7e8897  No.12631

>>12604

Thanks, this is pretty useful. Sankaku has an md5 lookup as well:


sankaku = open(fileName+"-sankaku.txt", 'w+')
sankaku.write("https://chan.sankakucomplex.com/?tags=md5%3A"+line)
sankaku.close()


af7c6a  No.12982

>>12604

>>12631

>>12602

I not able to get the md5 unless I open each image in the hydrus media viewer first, in which case there is no easy way to select them all and get it for all of them.


5942a0  No.13155

>>8396

Can someone bundle a torrent client with Hydrus+IPFS?

Just found this https://unix.stackexchange.com/questions/44247/how-to-copy-directories-with-preserving-hardlinks

(People in >>>/animu/81926 recommended Hard Links rather than soft links, so yeah)




YouTube embed. Click thumbnail to play.

0a8e64  No.13035[Reply]

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v357/Hydrus.Network.357.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v357/Hydrus.Network.357.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v357/Hydrus.Network.357.-.OS.X.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v357/Hydrus.Network.357.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v357.tar.gz

I had a great week fixing bugs and cleaning up other small jobs.

shortcut and hover window fixes

The recent menubar open/close detection did not work well on OS X and some versions of Linux. It misfiring broke some shortcut processing and hover window display. The good news is that as I went into that code more closely this week, I discovered several other menu, shortcut, and hover window problems that were causing long-term issues. All clients should have less janky hover window placement and behaviour, but particularly on non-Windows.

Specifically, the OS X shortcuts-in-media-viewer issue is fixed. Also, on all platforms, hover windoPost too long. Click here to view the full text.

9 posts omitted. Click reply to view.

f21949  No.13100

>>13035

I think pixiv updated their site again and now the downloader stopped working.


14c835  No.13101

>>13094

Yeah modified is fine, thanks.


aa61c3  No.13102

e621 did an oopsie with Cloudflare!


7c0ba9  No.13103

>>13100

Should be a fix coming in 358 later today! They switched up their API, and a user helpfully provided a new parser. Let me know if it still gives you trouble.

>>13102

Can you say a bit more about the error you got here? Was it the gallery page step, the file page, or the files themselves? I just did a quick test here on my dev machine and it worked ok here. I have seen it kick in site-wide for just an hour or so, which I guess are DDoS protection or similar, but I have also recently had several reports like this, where it works ok for one user but not another. I presume it is an automatic CF-side rule kicking in, based on IP or some other profile. Any chance you are on a VPN? If so, if you switch server, does that fix it?

I have a job to better detect the error page returned and giving a better 'CF problem' error and maybe 'ignored' status or something that can maybe try again several days later once I eventually have that download retry-later tech in. My guess is this problem will only increase in future.


aa61c3  No.13112

>>13103

Yeah CF got hit cause of muhChina, so it definitely is an issue for the meanwhile.




File: 8ac27dafade696d⋯.gif (435.05 KB, 593x539, 593:539, 8ac27dafade696d403d724ac76….gif)

df86ba  No.13098[Reply]

I had a great week working on the duplicates overhaul and fixing bugs. Potential duplicate pairs are not stored in a sensible way, with the duplicate filter now serving up more useful comparisons and clearing out the pending potential queue faster. Also, the issue with the media viewer not capturing mouse scrolls while not focused is fixed, as is the default pixiv parser, file repository file upload, and several more specific issues.

The release should be as normal tomorrow.



YouTube embed. Click thumbnail to play.

423e0a  No.12978[Reply]

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v356/Hydrus.Network.356.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v356/Hydrus.Network.356.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v356/Hydrus.Network.356.-.OS.X.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v356/Hydrus.Network.356.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v356.tar.gz

I had an excellent and busy week catching up after E3. Duplicate better/worse/same relationships are now stored more logically and efficiently, and a variety of bugs are fixed. Maintenance is also easier to control.

duplicates

The second phase of the big duplicates rewrite is now done! Rather than the old system of trying to store better/worse relationships for every possible member of a duplicate group, which is very complicated, requires many human decisions, and is not very useful in practise, this new system instead lumps all duplicate files into a single group with a single 'king' representing the best file of that group.

The grouPost too long. Click here to view the full text.

19 posts and 4 image replies omitted. Click reply to view.

da3139  No.13041

>>13025

>>13023

>>13022

You can import cookies, better help available on the discord


7e494b  No.13079

>>13013

Thanks. Yes, there will be a fix to seeing same alts again. Might be in this week as I do the main potential data, or in next couple of weeks if it proves complicated.


7e494b  No.13080

>>13019

>>13020

>>13021

Yeah, I'm going to try to fudge something like this. I'm not totally sure on all the edge cases yet, but I will do something. Maybe matching the smallest x or y to be the same virtual zoom height, with some clever exception for portrait vs landscape pairs. and offsetting drag delta by the (0,0) point on both files. I haven't gone into this exact code, so I can't speak cleverly yet, but it is in my big job todo. Thank you for your feedback.


7e494b  No.13081

>>13022

Yeah, hit network->downloaders->manage logins. There should be a gelb entry there by default. Add some credentials, set is 'active', and do a 'do login now' test. (it is worth doing a manual download as well there and then to test it worked for real–this stuff can be unreliable due to all sorts of factors). After then, if your login expires (I think gelb login lasts a year?), hydrus will try to login again automatically without you having to do anything. More info here, at the bottom:

https://hydrusnetwork.github.io/hydrus/help/getting_started_downloading.html


7e494b  No.13082

>>13026

>>13028

If you have help->advanced mode on, then clicking 'custom action' in the dupe filter should let you select 'alternate' as the dupe type, which then gives you the dupe merge options and a choice to delete A, B, or both. If you do it from the thumbnails, you should get the dupe merge options too.

I agree that having some sort of 'favourites' custom merge actions will be useful for alternate processing. I think I will do it when I get around to making a proper alternate processing workflow. I am trying to shy away from significant alternate processing in this primarily duplicates data structure overhaul, since it is a black hole of work that I could get lost in for months.

I took away the 'set as not related' from thumbnails because in data terms, this duplicate type is very inefficient if not applied to files that already have potential status. I have left it to the dupe filter and 'show some random pairs' quick workflow. I am still thinking about bringing it back for thumbnails, but at the moment probably won't. It really is a 'set as false positive' rather than 'dismiss any potential info' as it has been previously intepreted. I'm still thinking about it.




File: c047512639dafcf⋯.gif (365.06 KB, 300x300, 1:1, c047512639dafcf562b8e5fc24….gif)

14fce2  No.13030[Reply]

I had a great week fixing bugs and catching up on other small jobs. Some OS X and Linux UI issues (including shortcuts and bad hover windows in the media viewer) are fixed or improved, some downloader-page flicker is reduced, the client api is moved forward a little, and the popup system now has a neat collapse/expand button to hide subscriptions and so on while they work.

The release should be as normal tomorrow.

d5444c  No.13049

I'd like you to pay more attention to github




Delete Post [ ]
[]
Previous [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15]
| Catalog | Nerve Center | Cancer
[ / / / / / / / / / / / / / ] [ dir / b2 / b4 / choroy / coz / fanfic / fascist / marxism / radfem ]