/hydrus/ - Hydrus Network

Can someone explain where this DB is obtained from?

>>230

wait, why remote instead of local? isn't that only if you're using hydrus as a server?

>>1930

the stickied database thread (PTR sync) has all of them

Does anyone know why most of my files aren't tagged? I think I have about 22k in images and maybe 1k are tagged and I have local danbooru and other Dbs set up from what I can tell. restarting doesnt do anything anymore.

>>1946

yes. all of them are checked. to give a kind of example. I may have exaggurated the tag count being a little lower, but in general, its too low to make sense. i have about 7+k untagged completely, about 10k more are ones i've personally tagged as either nsfw or vore, so that I can easily sort images, which means about 8k (conservative estimate) of that are untagged as well other than those two.

Thing is, a lot of these images i know i;ve gotten off places like derpibooru etc, where I know theres a duplicate.

Is there no way to set the whole thing to passively strip metta data image by image from these places like happy panda does? it seems like that would be the easiest way to go about it, even if it would have to be slow.

>>1949

>about 10k more are ones i've personally tagged as either nsfw or vore

Did you add these tags to the files metadeta directly? Because that would change the hash of the file, which would explain why it is missing some.

I accidentally made that mistake while using Directory Opus. Removing the metadata should restore the hash back to normal.

>Is there no way to set the whole thing to passively strip metta data image by image

It doesn't. That would be a good idea though to request in the suggestion thread. For the time being, there should be plenty of tools/programs that can bulk strip metadata from files. If not, then I can write up a quick script if you want.

>>1963

it was through the hydrus system, so if so, then yes, but if it doesn,t then no.

if you do come up with a script that does that, that would be good i think. I have a hard time believing it will fix the issue, but any tags can simply be reapplied anyway thanks to the way hydrus works.

>>1964

If it was through hydrus then the file's metadata wasn't touched.

Can you post 3-5 images you have from gelbooru that weren't tagged?

Also, did you use gelbooru or gelbooru2016? The gelbooru2016 one should detect all images up to Jan 15th

>>2026

oh, and now that i think about it you said adding tags in the program didn't count anyway, it's been a while. that would actually be stupid if it did now that i realize it. but the point remains, these images didn't get any tags despite me having both gelbooru 2016 and derpibooru

>>2027

i dont know about the other images, but the asanagi image doesn't match the same hash as from gelbooru. Not that anon, but its probably a bad idea to add images which came from other sources(4chan/8chan/etc)

>>2029

Right, so on that note, is there any easy way to get the image that the booru's would sync with in mass? As in, say i have 3k photos from halfchan, 8chan, etc.. but i downloaded the directly off the site instead of reverse searched and getting it off the booru.

Would there be any way of getting all the images i have reverse imaged searched and a booru one selected in its place? It wouldn't be very practical to do it by hand, but if it's what must be done…

>>2030

At minimum if it cant be done that way, then theres the potential for a halfway step, something like a program that searches for like images in these databases, then provides a link directly to the image for download.

Ultimately though, this seems like a vital problem that needs fixing in the main program, maybe an image hash corrector, or something to outright replace the file, once you check that it corresponds with the ones online with a maximum ammount of certainty.

The reality is image boards is partially why this is being used at all, so not having any compatability with images saved from boards is a problem.

PLEASE READ THIS THE MOST THOUGH

The images I posted came directly from the booru sites. the ones selected have not been downloaded from 4chan. so while all that above is a good idea, I don't suspect that it's as simple as which website it was downloaded from.

>>230

Where did you get hold of those furaffinity db's? Any way I can grab a copy?

>>2057

anyone?

>>2057

>Ultimately though, this seems like a vital problem that needs fixing in the main program, maybe an image hash corrector, or something to outright replace the file, once you check that it corresponds with the ones online with a maximum ammount of certainty.

Duplicate image checking is one of the things the dev has on his backlog, and that might be able to help with what you're describing, too.

>The images I posted came directly from the booru sites

Are you sure you downloaded the correct version of the image? A lot of boorus will display a shrunken version on the page if the original is too large. This shrunken image will naturally have a different hash than the one the db is using. Make sure to click "download" or "original" or something similar to get the right one.

That might explain why some of your images from chans aren't working, either. Dumb anons uploading the downsized samples instead of the original

>>2119

Why the hell do boorus even do that, instead of having the image scaled down in js? Saving on bandwidth?

>>2119

>Duplicate image detection

isnt this merely for files within the database? how will that help?

>>2122

probably.

Anyway, even something simple as searching for similar images and then providing a list of links to where they can be found to resave would be better than finding it on our own. it'd cut down on the time tremendously and be in line with the purpose of the program.

>>2123

>isnt this merely for files within the database? how will that help?

>someone ends up with both versions

>he detects that they're duplicates

>makes sure to give them both tags

>your duplicate now has tags

>Anyway, even something simple as searching for similar images and then providing a list of links to where they can be found to resave would be better than finding it on our own. it'd cut down on the time tremendously and be in line with the purpose of the program.

On the contrary, I feel like that would actually go outside the scope of the program. You or someone else should make your own program meant exclusively for searching for the best quality image instead of bloating hydrus further.

>>2124

>>they make sure both tags have duplicates

or they delete them and save themselves the work

>>on the contrary, something that would help accomplish the goal of creating a public image tagging database, should not have a system to deal with 80% of images saved by users

are you serious here? you realize this is meant to be used by people and not robots yes?

that you can only benefit from this so people tag the right image and the metadata is shared to everyone? Hydrus is far from bloated.

>>2127

>Hydrus is far from bloated

The 20 nonstandard python libraries beg to differ.

The dev manages to keep it all fairly optimized and running smoothly, I'll give him that, but hydrus does have a lot of feature creep and we're definitely heading in a bloated direction. Dev is too charitable to us with all our feature requests :)

Because of the library situation though, I fear that hydrus will be broken if the dev ever stops maintaining the code. By nature of python, libraries will update and change syntax, and then us non-Windows users will need to start keeping older versions of python libraries around to continue running hydrus… I should formally request to the dev that he create a "module" concept: example- I shouldn't need to have python-socks installed if I'm not making use of the proxy feature.

Anyway, you don't sound very technically minded because you just requested that hydrus shit unicorn balls, and passed it off as if it were no big deal. The idea of your feature might fit neatly into your understanding of hydrus's scope, but the scope is and should be tighter than just "media storage and sharing"- there are programmatic limitations too. If the idea would take a ridiculous amount of code and effort to implement, it probably needs to be its own separate project.

There's really no way to do what you asked, unless you want hydrus to automate sending thumbnails to saucenao/tineye/google/bing reverse image searches. Which would be an awful hack of a feature and require constant maintenance and never work very well in the first place.

What you CAN do is start collaborating with other users on creating a database of every crappy image & higher quality source picture. Database could contain hashes of the bad images, a way to promote the best image associated with that hash, and return the ipfs multihash of said best image. Hydrus could then programatically hook into that: user sends a hash to your database service, gets back ipfs multihash of your best available copy (and maybe an http:// url to try if ipfs fails), and attempts to download it.

btw,

>>posturl

>text quote

>>2130

A stand alone program that checks an image similarity is not unicorn balls. Incorporating it into hydrus would probably be even less work depending on how it's designed. by the sound of libraries it probably would be less work. Checking a file against an image similarity database automatically, such as iqdb, and pasting the link to those places so the user can mass download the correct hash is a near essential function if the program is not going to be able to properly identify the throngs of images that are similar.

Your second complaint, that it is beyond the scope of the project, is essentially meaningless, because the entire project is beyond the scope of the project. Hydrus arose on the chans to support the organization of images, images that would no doubt incorporate ones shared across images boards, never-mind the apparent issue with files saved directly from a booru not being tagged. If it cannot do even a vaguely better job than it does, there is no purpose in using it, because it wont perform it's most basic function; tag the same images communally so people don't have to reinvent the wheel.

This is a direct obstacle to the functionality of the program, not a bid for a special embellishment.

>>2137

Geez, your posts really scream of entitlement, if I do say so myself.

>A stand alone program that checks an image similarity is not unicorn balls.

Your idea might sound easy to you, but I can tell you it's a lot more complicated than you think.

Besides, it's not the job of Hydrus to get you the best quality images it can. It's also not the job of Hydrus to make sure that every image you save is already tagged (those booru databases aren't even made by the dev.). For me, the core functionality of hydrus is to be able to organize my images by tags, and to retrieve these images later. Anything else is bells and whistles. The PTR is nice, but it's not the core feature of the program. When it does tag something I save, it's nice. But I almost always end up being the one to tag my images anyway (or rip them from the booru I saved from). And that's ok.

Rather than asking the dev for the world and more, a better option would be for Hydrus to allow for other programs to interface with it with it, somehow. Let someone else make the duplication checker. Let someone else create a quality check db like the other anon suggested.

If you want to make sure your images get tagged, then make sure you get the right hash. Use a downloader program (or hydrus's own download page, which will rip tags for you). If you see someone on a chan posting a sample, call them out on it.

>>2138

>>Your idea might sound easy to you, but I can tell you it's a lot more complicated than you think.

it isn't.

>>Besides, it's not the job of Hydrus to get you the best quality images it can.

This is only a work around because you are suggesting that syncing the hashes is impossible without redownloading the images. which would be even more intensive on hydrus than the compromise I worked up.

>It's also not the job of Hydrus to make sure that every image you save is already tagged (those booru databases aren't even made by the dev.).

Not every image, but certainly enough for it to be useful. the fact is, currently. Hydrus's tagging system offers so little communal support because of image variation, that it might as well not exist, and people should start from scratch.

>>and thats okay

there are hundreds of programs that do that better. about the only thing hydrus has in it's favor as far as I know is the copy path function.

>>The PTR is nice, but it's not the core feature of the program.

it is a core feature. it is one of the principle design elements of the program. the author specifically desires to avoid reinventing the wheel for their tagging by having it be communal. that is what hydrus does that others don't.

>>Rather than asking the dev for the world and more

this is, again, a core feature of the program.

>> a better option would be for Hydrus to allow for other programs to interface with it with it, somehow. Let someone else make the duplication checker. Let someone else create a quality check db like the other anon suggested.

if someone can do this, then great. plug ins are pretty standard in many programs

>>2138

>>make sure you get the right hash

pray tell, if downloading something directly from a booru does not get me the right hash, what the fuck do you, oh holier than thou one, expect for me to do?

Downloader programs, especially hydrus's, work on a subscription model by the way, and I do not download anything near that amount of images by category only.

>>2139

>pray tell, if downloading something directly from a booru does not get me the right hash, what the fuck do you, oh holier than thou one, expect for me to do?

I already explained you're probably doing it wrong. You can't just right click-> save image. You have to click download, or get original image, and save that. And if the databases used the wrong image too, that's their fault and should be fixed.

Also most boorus will let you get an individual image using the md5:<hash>. It's a hassle, but the image's filename on the booru will be its md5 hash, so it's not hard to get.

>>2140

if you are referring to making sure the images are not samples, I am aware of that, in case that hasn't been made clear.

I don't understand what the second line is however. do you mean pasting the filename into the booru file path?

>>2141

If you're aware of it, then I'm not sure why you're still getting the wrong hash.

For the second line, here's an example.

Say you want this image: http://gelbooru.com/index.php?page=post&s=view&id=3073376

If you go to the location of the original image, you get http://simg4.gelbooru.com//images/d3/b1/d3b10041185a46c75027f9a164a3e9f1.jpg

d3b10041185a46c75027f9a164a3e9f1 is the image's md5 hash.

you can search md5:d3b10041185a46c75027f9a164a3e9f1 on gelbooru, and the result will be just that image, like so

http://gelbooru.com/index.php?page=post&s=list&tags=md5%3ad3b10041185a46c75027f9a164a3e9f1

so if you put that search into hydrus, you'll get just that image and its tags. Like I said, it's a hassle, but it works.

The better way would be to have a browser plugin that lets you save the tags in a text file which hydrus could then import, but until I learn of such a plugin that's how I import individual image tags.

this doesn't on every booru, mind.

>>2142

neat trick- that might have allowed me to recover some of my missing images if they allowed sha searches. ah well.

>>2139

I am completely lost. So the feature you want, which is definitely beyond the scope of hydrus (please just accept that, at the very least, if it requires a huge time investment, it should be its own separate project), is really just a workaround for… what exactly?

>>2137

>a stand alone program that checks an image similarity is not unicorn balls

Hey, don't rewrite history there. You were requesting this be done over the internet, which is an entirely different ask, and requires cooperation from an existing external database and an API to hook into. Unless you can come up with an exact specific methodology that proves us all wrong about how complex this request is, the not-dick move here is to ask more questions so we can better explain away your misunderstandings.

>>2149

>, which is definitely beyond the scope of hydrus (please just accept that, at the very least, if it requires a huge time investment, it should be its own separate project),

you being willfully illiterate and oblivious doesn't change the purpose of hydrus and it's shortcomings for its base function.

Read it again: the work around of generating links to the correct images with the tags through an online service, is to avoid needing to build a way for hydrus itself to detect if an image is similar to another with tags and have the image take the place of that hash in the database.

>>2150

>>don't rewrite history here

this is not rewriting history, what I described already exists outside of hydrus.

>>the not-dick move here is to ask more questions so we can better explain away your misunderstandings.

I am not acting like a dick, nor am I misunderstanding here. Hydrus automatically downloads images from websites based on certain metadata. it is only one step further to submit that data instead, and record different data, namely links instead of images. Nor would it be troublesome to tell hydrus what data to submit.

Okay, lemme just try and break down what HELP is asking for.

1) You've imported an image. This image's hash does not currently exist in the PTR, or if it does it hasn't been tagged. An identical or higher quality image with a different hash, however, does exist and is tagged.

2) Hydrus should detect that you have an inferior duplicate. Duplication checking is currently done with their hashes, so I'd think this should be doable even with images you don't own. Determining the inferior automatically is what makes this part tricky.

3) Hydrus should then give you a link to the better image, assuming such a link exists.

Honestly, I think this whole idea breaks down as early as 1). There are lots of images which aren't tagged yet. HELP wants nearly every single image he imports to come pre-tagged, but that simply isn't going to happen (barring integration with that AI tagging software).

Part 2 is the most feasible part of the idea. Duplication checking exists, and you could use the PTR to find similar images. But like I said, you couldn't automatically determine which one is superior, at some point you need manual oversight (image sets with only tiny variations between each image being a good example).

Part 3 is where the idea breaks out of the scope of Hydrus, imo. You can't search for an image using its SHA hash, to my knowledge. You have to search by the image, which means someone has to maintain a 3rd party database of these source links. That's a decent amount of work on its own, but consider what it would take to retroactively source the 20+ million mappings already in the PTR.

And the above assumes a link exists, which is a fairly big assumption. Not everything is in a public, easy to find booru. There are still many, many images which can't be found through a tineye or google image search.

>>2154

Oh, but once IPFS gets working, that last bit becomes moot. If all we need is the IPFS hash of the image, then it'll be easy to get links to the image.

>>2154

>Honestly, I think this whole idea breaks down as early as 1). There are lots of images which aren't tagged yet. HELP wants nearly every single image he imports to come pre-tagged, but that simply isn't going to happen (barring integration with that AI tagging software).

apparently you missed the fact that these images I have that aren't tagged do have duplicates from sites that are added to the database, so you're wrong.

>Part 2 is the most feasible part of the idea. Duplication checking exists, and you could use the PTR to find similar images. But like I said, you couldn't automatically determine which one is superior, at some point you need manual oversight (image sets with only tiny variations between each image being a good example).

the one that's superior would be the one with the most tags. and it would be easy enough to find them even without hydrus's input, as online services exist to match images to boorus such as iqdb, and can be used in place of hydrus. the creation of the link is the easiest part, since such services automatically provides them and hydrus already has the ability to take information from a web page based on metadata.

>Part 3 is where the idea breaks out of the scope of Hydrus, imo. You can't search for an image using its SHA hash, to my knowledge. You have to search by the image, which means someone has to maintain a 3rd party database of these source links. That's a decent amount of work on its own, but consider what it would take to retroactively source the 20+ million mappings already in the PTR.

as I mentioned before, the work is already done for most booru. I never said anything about searching by hash either. We know that Images with the hash we want are the ones served from sites like danbooru and gelbooru because we know we are using their scrubbed data to tag those specific images. The link would be provided by online service. Hydrus would only need to upload each one and take the resulting links, which is a tedious and long process.

I admit I have no idea what IPFS is, but if that would work too, thats great, but what I'm proposing is more like a website image description that requires input as well as receives data than the creation of a huge database. Particularly, Hydrus is not a downloading service (images are acquired from 3rd party places) so there's no reason not to vet image hash's from third party sites as well, it will probably be far slower that way but it should make it far more feasible to program.

>>2156

I suppose as an afterthought. if you wanted to, I suppose you could record those URLs and associate them with the undesirable hash to turn it into that service. that would, eventually, create that database, but that's beyond the scope of what I'm suggesting.

>>2156

>apparently you missed the fact that these images I have that aren't tagged do have duplicates from sites that are added to the database, so you're wrong.

In which case this is isn't an issue with Hydrus itself, but the databases. Complain to the guy making them to fix it, instead of asking Hydrus to add more features.

>the one that's superior would be the one with the most tags

That's not necessarily true, though. What if a smaller resolution image were the one that was tagged? Or it was part of an image set, where each image is similar, but different enough to need different tags?

>as online services exist to match images to boorus such as iqdb, and can be used in place of hydrus.

If all you want is a way to search for the image through hydrus, that doesn't sound too infeasible. Make a search request with the image, look for results that are boorus, and give back the tags for the image. That seems much more reasonable.

>We know that Images with the hash we want are the ones served from sites like danbooru and gelbooru because we know we are using their scrubbed data to tag those specific images

We actually don't know this. Again, those databases were created by a third party. I could create my own database right now full of arbitrary tags, and Hydrus would handle it the same way. Neither hydrus nor those databases know anything about where the mappings come from.

>>2158

>In which case this is isn't an issue with Hydrus itself, but the databases. Complain to the guy making them to fix it, instead of asking Hydrus to add more features.

wrong again, the problem is with the images, as discussed earlier in the thread. the databases cannot possibly be made to have hundreds of hash duplicates, it makes far more sense because its an order of magnitude less of repeating drudgery work tagging things to simply use one hash and to replace the undesirable images with the correct ones.

>That's not necessarily true, though. What if a smaller resolution image were the one that was tagged? Or it was part of an image set, where each image is similar, but different enough to need different tags?

As this replacement is still done manually the second issue is never going to be a problem, you are inventing excuses at this point. the service provides links to the hash that has tags based on booru. that is all. as for if a site uses the smaller image, that situation seems so rare that I have a hard time believing at this stage in development it needs to be accounted for, so while it's plausible, scrapping an entire direly needed feature for it doesn't seem relevant

>If all you want is a way to search for the image through hydrus, that doesn't sound too infeasible. Make a search request with the image, look for results that are boorus, and give back the tags for the image. That seems much more reasonable.

Grabbing the tags is a step above what I had planned. I simply intended for hydrus to submit the images over time, and retrieve the links the site iqdb provides. If the databases must tag specific hashs, having a link to those hashes should provide the tags anyway.

>We actually don't know this. Again, those databases were created by a third party. I could create my own database right now full of arbitrary tags, and Hydrus would handle it the same way. Neither hydrus nor those databases know anything about where the mappings come from.

ther ear two outcomes here, because this sentance is slightly confusing. Either you mean that these databases cannot reliably keep the hash identical to the booru, in which case databases are fundamentally flawed (I'm going to assume this isn't the case) OR, that the hashes themselves are arbitrary becuase databases are, but if that's the case, for our purposes, it shouldn't interfere as long as the hashes remain the same and therefore can be identified by hydrus as those images.

>>2159

>wrong again, the problem is with the images, as discussed earlier in the thread.

If you know the images exist on a booru, then either you're saving the wrong images, or the database is. I'm not sure what's so hard to understand about this. In case there was miscommunication here, when I say "database" I specifically mean the ones from this guy >>1619, not the PTR

>I simply intended for hydrus to submit the images over time, and retrieve the links the site iqdb provides

what the fuck is the point of even integrating this into hydrus if that's all it does? Just make the searches yourself. If scalability is the issue you could make a script of some kind, but otherwise I can't see that being nearly useful enough.

>Either you mean that these databases cannot reliably keep the hash identical to the booru

Most boorus use a different hashing system than Hydrus (md5, while hydrus uses an SHA hash). There's no way to know where an image came from based on its hash. While we know those databases are scrubbed from boorus, Hydrus doesn't, and there's no metadata in those databases to tell it that.

>>2153

>you being willfully illiterate and oblivious

and this is where I stop reading.

We have an adolescent twat in our midst. I will not be dignifying it with attention, I suggest the rest of you also ignore it.

>>2184

>I'm curious though, what's the need for un-namespacing tags

I think I can see the reason behind it. For example, Rule34Paheal & Rule34Hentai don't use namespaces for tags. So let's say you wanted all images from character:elin.

You wouldn't get any results from those two sites, since they don't use namespaces. You'd have to search for the un-namespaced elin. However, since some sites DO use namespaces, you also wouldn't get any results from typing in un-namespaced elin from booru* sites. The only way to bypass it is to make 2 separate searches, or of course recreate the entire db without them.

I believe OR logic was already in the poll, so it'll get implemented eventually. That way, you can do character:elin OR elin, thus bringing up results from both in a single search. I'm not sure if that's why >>2178 wants to do it, but that's why I'd see the reason for doing it.

>>2185

Or an even simple method for this case would be a button/syntax for automatically searching unnamespaced tags, since OR logic seems much more complex.

So typing 'character:elin',would automatically give results for unnamespaced elin too. Or '+character:elin', to save on gui space.

>>1932

And I just realized why

Putting a tag archive in remote -> tag repo let's you search for tags from that specific tag service only. Before I was running searches on local tags, which meant every tag archive I had synced. Before, I always had assumed it was related to running a server, so I never bothered checking it out. The more you know~

>>2184

>>2185

>>2187

I just don't use namespaces, don't see much point in separating them from the other tags as namespaces when I'm only going to be typing "Yui Hirasawa" in anyway.

The rule34 sites is a good reason however, and the boorus have eastern naming conventions for some of the files I've tagged (last name first), so anything I manually tagged would be in the opposite order and I would have to do searches.

Is there a different approach to doing this, some sort of alias setting? I could use that in other tags; I've got K-On! as a tag, but I'll always type "kon" because who would drop the hyphen in there on a search? Windows would ignore the hyphen and display files with K-On! as a tag when you search "kon" for example.

>>2185

>However, since some sites DO use namespaces, you also wouldn't get any results from typing in un-namespaced elin from booru* sites

Actually, if you search for an unnamspaced tag, hydrus will return all instances of that tag, ignoring namespace. I know this from experience (A lot of pokemon images are tagged with the character namespace, while others use the species namespace. If I want both I have to search for the un-namespaced pokemon name).

>>2190

>I just don't use namespaces, don't see much point in separating them from the other tags as namespaces

It's pretty useful when the same tag might mean different things in different context. A "series:archer" tag means something very different from a "character:archer" tag, which in turn is very different from a generic "archer" tag.

>The rule34 sites is a good reason however, and the boorus have eastern naming conventions for some of the files I've tagged (last name first), so anything I manually tagged would be in the opposite order and I would have to do searches.

This is the exact situation tag siblings were implemented for. Look up in the helpfiles what those do and how they work.

>>2191

Would the tag siblings thing be applicable to the Keion example?