/hydrus/ - Hydrus Network

Here is an update, up to version 159 and update 1107, which is today, June 10th 2015. It now has 14.2 million mappings.

http://www.mediafire.com/download/hf7c5bvxpe7uz65/Hydrus_Network_Client_bare_database_with_PTR_up_to_update_1107.7z

Also, here is a tag archive of the same:

http://www.mediafire.com/download/0ycnnw1yooo0z2w/PTR_HTA_up_to_update_1107.7z

And another, up to version 176 and update 1212, which is today, October 9th 2015. It now has 20.5 million mappings.

http://www.mediafire.com/download/df4bs537sih4aux/Hydrus_Network_Client_bare_database_with_PTR_up_to_update_1212.7z

And the tag archive:

http://www.mediafire.com/download/d2j6prlr7ov5840/PTR_HTA_up_to_update_1212.7z

>>1263

Just the db. Swap out the db directory with the 'bare_database' 7z, and you should be good to go.

This is a bare database, though. It is essentially a fresh install that has synced to my public tag repo and done nothing else. All the options and files already in your client will disappear, so you will have to reset and reimport them, which might be more than a new user wants to deal with.

If you want to stick with what you already have, you can change when and how the client does its high-CPU processing in file->options->maintenance and processing. If the client is processing and freezing up when you don't want it to, let me know.

>>1436

Yeah, they are the same, just in different containers. The archive is really just for programmers who want to screw around with the data alone.

Finally got around to playing with tag archives thanks to an anon's scripts. I have some questions though.

1. I made a tag archive of xbooru. It has 556,667 hashes. Does this mean Hydrus should be able to recognize 556k images from xbooru once imported? As in, I shouldn't need to run the downloader in order for it to grab tags for the image.

2. Would having too many tag archives slow or clutter my client.db? I currently have gelbooru, danbooru, e621, derpibooru tag archives imported.(In progress of importing rule34xx & xbooru..here's hoping the import is successful) In the future I plan on doing more, such as rule34phael & rule34hentai, so I'm curious if everything will still hold up. I don't know if I'm the only one taking full advantage of these, so i'm asking to make sure.

3. This is kind of unrelated, but is it possible to sync my client.db with the PTR? The main reason I'm making the tag archives is because I'm not sure what booru siterips PTR already contains. Hopefully I can get majority of them in the PTR.

>>1585

1

Yes. Any files you already have that match any of those hashes will gain the namespaces you chose in manage services. Any files you import in future will be checked against the tag archive in case they match as well.

2

The only time the client queries a tag archive is on the initial sync and the check every subsequent file import. It typically only takes a couple of milliseconds per import per archive, so don't worry about it–add as many as you want.

3

It depends on the hash type of your archive. The hydrus client and server refer to files using sha256, but I assume xbooru has provided you with md5. The hydrus client only has md5s for local files, so it can only cross-reference to make the conversion of (md5->tag)->(sha256->tag) for files you actually have. As you have probably experienced with your other tag archives, this still usually represents a hell of a lot of tags!

If you happen to have sha256, though, then you can potentially sync the entire gigantic tag archive into the PTR! This is a slightly terrifying prospect, but if you can, give it a go! I expect trying to upload 9 million mappings or whatever in one go will cause an error in one way or another, so let me know if you end up in this situation–I might have to rewrite the upload system a little to spread it out a bit or add a cutoff or something.

If you do have md5 or sha1, you can get better coverage by sharing the archive around many users, who will have many of the files that you don't, enabling a greater number of md5->sha256 conversions. Would you be willing to share your archive? I would 7zip it and mirror it up on my mediafire like the others.

I apologize. I had to delete my xbooru.db download link. Although it does work, it seems my xbooru namespace.csv was missing a bunch of namespaces. (could have sworn I changed the max Pid) As you can see in the picture, it did match the md5 with the proper tags, but "isabelle (animal crossing)" should have had the 'character:' namespace. I'll put the link back up when it's finished.

>>1594

Some updates.

Rule34xxx tag archive is done

xbooru tag archive is done

Rule34Pheal tag archive is done

sof.booru.org tag archive is done(*1)

ohdd.booru.org tag archive is done(*1)

I will post all of them once they're done syncing.(or on my off day) Everything seems to be going fine though.

(*1) Might be interesting to some, but I discovered that some of the sites from booru.org(sof/ohdd,etc) use incorrect hashing. Initially I thought they were using SHA1 since they were 40 characters long, but then noticed that none of my files were being recognized by hydrus, even after using HASH_TYPE_SHA1 to generate the db. Turns out, the hashes were completely wrong, and were a mixture of SHA1 + MD5. This meant converting all hashes in the csv to md5. Here is the script I made to convert the hashes.

#USAGE: md5-convert.py "directory_with_files" ".csv file"

import hashlib
import codecs
import sys
import os
import re
import time

DIR = sys.argv[1]
CSV_FILE= sys.argv[2]
count = 0;

#gets md5 hash of file
def md5(fname):
    hash = hashlib.md5()
    with open(fname, "rb") as f:
        for chunk in iter(lambda: f.read(4096), b""):
            hash.update(chunk)
    return hash.hexdigest()

f = codecs.open(CSV_FILE, encoding='utf-8', mode='r')
data = f.read()
f.close()

with codecs.open(CSV_FILE, encoding='utf-8', mode='r') as f:
  for line in f:
  
	line = line[:-1]
	line = line.split('\t')
	ext = re.search("\.jpg|\.jpeg|\.png|\.gif|\.webm", line[2]).group(0)
	hash = line[1]
	
	#construct absolute path for the md5 function
	path = sys.argv[1] + '\\'+ hash + ext
	
	#check to make sure the file exists
	if os.path.isfile(path) and os.access(path,os.R_OK):
		filehash = md5(path)
		newdata = data.replace(hash,filehash)
		
		#write data to temp file
		f2 = codecs.open('temp.csv',encoding='utf-8', mode='w')
		f2.write(newdata)
		f2.close()
		
		#set old data to the newdata
		data = newdata
		count+=1		
		print "line #{count}  converted to md5.".format(count = count)
		#time.sleep(.250)
		
print "All hashes have been converted."

http://pastebin.com/TdakdrpN

After converting and recreating the db,they now work fine.

>>1599

That mixed hash thing is odd. Where are you getting the incorrect 40-character hash from? I tried gelbooru's api call:

http://sof.booru.org/index.php?page=dapi&s=post&q=index

http://ohdd.booru.org/index.php?page=dapi&s=post&q=index

but it just sends me to blah.booru.org//. Yet it works for SizeBooru, here:

http://size.booru.org/index.php?page=dapi&s=post&q=index

Am I doing something wrong there? Maybe those boorus just have the API turned off.

By the way, if you want to generate sha256 for those files, it works exactly the same, just use hashlib.sha256(). If you are interested, I use it because modern CPUs can generate it quick and it isn't easy to generate a collision. md5 is easy to spoof:

http://www.mscs.dal.ca/~selinger/md5collision/

And as I understand it, sha1 isn't far behind.

>>1608

>sha1 of the literal md5

lol wat

I can sympathise, though–I bet the dev wanted to move to sha1 but didn't want to recalculate everything or something, or he thought sha1( md5( file ) ) was the same as sha1( file ). I see that 0.2 goes back to something more reasonable.

I have come to like python in a not dissimilar way. It is a bit janky sometimes (GIL is almost a deal-breaker, imo, and should have been fixed years ago), and it runs a bit slow, but I find I can prototype things so much faster than if I do them 'properly'. Since I am always short on time, fast development is a high priority for me. I think they should teach it in school, as any smart kid, given the python console, can be up and running in about fifteen seconds. Instead it seems they often go for some opaque bullshit like Java.

Hydrus indexes all the hashes it stores, so it is about the same speed for any of them. The smaller ones are probably quicker just because there are fewer bytes to compare, and the db index is smaller because it only stores local files' hashes, but I expect the difference is measurable in microseconds.

sha256 is a good bet for anything hydrus-related just because it is the native hash, so you can transfer it over the network. I knew when I chose it that it would be a small problem interfacing with the legacy formats most other sites use, but I am spergish enough that I would rather do what makes engineering sense to me than work with consensus. Then again, someone was telling me IPFS uses some metahash or something where it basically adds a hash_type descriptor-header to its hashes, like "sha1:[sha_hash]", which mite b cool to support in the future, especially if it turns out we all need to move to SHA3. I would rather that the booru/imageboard devs just jumped to sha256 for now, though.

>>1611

Well here are the tag archives.

Hopefully you can mirror them to the hydrus mediafire folder too. They should be complete tag rip, as of Dec 10~20ish. Some of the sites do not use namespaces, so you'll get some tags creators/characters/series as generic tags. I couldn't think of a way to fix that without manually editing the tags(yea..not going to map out 1mil tags lol) or using another sites namespace, which could technically work, but it'd probably be messy and miss a bunch of tags too, especially if it's gelbooru's namespace since they use japanese names.(For example, rule34 uses the name "Dawn", yet Gelbooru uses "Hikari", meaning all pokemon characters wouldn't be mapped if that tag was found)

Link: https://mega.nz/#F!HUIX1ZIB!buAIE_bgKCKZ0G0Mjk4CGQ

db: ohdd.db

name: Onahole Doll - オナホドール

site: http://ohdd.booru.org/index.php

db: rule34hentai.db

name: Rule34Hentai

site: http://rule34hentai.net/post/list/1

(includes all the loli tags too)

db: rule34pheal.db

name: Rule34paheal

site: http://rule34.paheal.net/post/list/1

db: rule34xxx.db

name: Rule34xxx

site: http://rule34.xxx/index.php?page=post&s=list

db: sof.db

name: Semen on Figures

site: http://sof.booru.org/index.php?page=post&s=list

db: xbooru.db

name: Xbooru

site: http://xbooru.com/index.php?page=post&s=list

Pic-related is an import of some rule34hentai webms I saved, being tagged accordingly.

>>1619

>>1620

This is fantastic, thank you for the hard work!

I am uploading them to my mediafire now:

https://www.mediafire.com/folder/yoy1dx6or0tnr/tag_archives

>>1668

Update.

Here's the next batch that I will release.

Gelbooru [2016] - Done/Synced

Safebooru - Done/Synced

Dollbooru - Done/Synced

Uberbooru - Done/Not synced.

Nihonomaru - Done/Synced

My Figure Collection - In progress. Should be done in a week or so. The problem is the site does not store any of the image's hashes, and they also have a page limit like Sankaku. This meant using the dirty method..(iterating through every id one by one(don't want to spam the server)), and it also meant I have to download every image off the site to get their hashes and use them in a .db, which is roughly around 1.5mil. Not a huge deal, just takes awhile. Once done, it should also support the proper namespaces the image belongs to.(Figures/Items/Collections/etc). Pic-related.(The '00000' column will be replaced with sha256 of the files once they're downloaded)

Sankaku Idol/Sankaku - In progress. I found a way around their page limit, so I should be able to make a .db of it.(Same method as MyFigureCollection)

>>1820

1. Shut down Hydrus if it's running first

2. Drop the .db in '\db\client_archives', and start hydrus up again.

3. Goto services -> manage services

4. In the local tab under 'local tags', click the add button under archive sync.(Pic-related)

5. Then check all the boxes for tags you want to sync.(no namespace/character/artist/etc)

Depending on the size it can take quite some time. Once synced, hydrus will be able to tag any file that came from that site, up to the date the archive was created.

> grab the drawfriends and deviants depository boorus

Sure thing

>>1820

Alright, here are the ones you wanted, tested and working. Since they were small, I could finish them quickly.

drawfriends

https://mega.nz/#!KJhE0YQS!ez8HoC-LS4mZB-xW5-phcTZahYFX6dGGzBehkd_tZco

deviants-despository

https://mega.nz/#!vB52CRxZ!lenOJaEAWsp2ovMSU3UoDG3iuZB0s0PXQhNekwdghpI

That's a lot of archives, thanks.

But what should i do if i already synched my db with old gelbooru archive? Should i delete it before adding new one?

Archives downloaded from:

https://www.mediafire.com/folder/yoy1dx6or0tnr/tag_archives

Tested & Working Archive Tag Packs:

rule34phael

rule34.xxx

drunkenpumken.booru.org

tbib

xbooru

lolibooru-moe

Broken or malformed Archive Tag Packs:

Danbooru

Gelbooru

(See image)

Danbooru's fails to load

Gelbooru's has a bunch of extra tags that may or maynot cause an issue (IE: Only select Character, Artst, Series, Creator instead of the misspelled ones).

If anyone can confirm Gelbooru's status that would be great.

>>1833

awesome, thanks friend, these two have a lot of tags for some more obscure images

>>1204

And now up to version 191 and update 1310, which is today, January 30th 2016. It now has 33.7 million mappings.

EDIT: The previous link was lacking client_updates and the other folders, here is a corrected upload:

http://www.mediafire.com/download/t260gvff3ytp2w2/Hydrus_Network_Client_bare_database_with_PTR_up_to_update_1310.7z

Link:

https://mega.nz/#!SZR2GaSY!0OzfLhBPO0EHsqF4py8jk3m39mVCe6E7UYfYHw9Itq8

(edit, some series weren't being recognized because the tags had colons in them. I realized I could use this to my advantage and add "series:" namespace to them, so if you downloaded the .db 3-5 hours ago, then please use this instead)

db:myfigurecollectionV2.db

name:My Figure Collection

site:http://myfigurecollection.net

Finally done. This is a tag archive of the pictures section from MFC, which includes over 1.3 million photos of figurines.

In Hydrus, You can search by Category(category:figures, category:kits&customs,etc), image ID and the uploader.

There's two portions to the site. One where users upload images of photos they've taken(the current tag archive), and another portion where the creators sell their figures, which has more detailed information such as the company, character, materials,etc. That part will be done in a few days.

Hopefully anyone else who is interested in figurines will benefit.

>>1909

Glad I could help.

Link:

https://mega.nz/#!DE5EUaYA!lQkqVE6_35m26XFQJs8TH_O2P6lS_U-vzh2JM4xnaZQ

db:myfigurecollection-item.db

name:My Figure Collection

site:http://myfigurecollection.net/search/

Here is the second portion of MFC, the item database. You can use it with the mfc picture db to find a bit more information from a figurine. You can search most of the same things you would from the myfigurecollection.net/item/ page. Basically, same as using the site, except in Hydrus. Don't forget to add colors to all the namespaces added.

list of item namespaces;

origin

company

creator(both sculptor and illustrator)

character

material

id

uploader

category(prepainted, action/dolls, garage kits)

scale

As for Sankaku Idol/Chan..Those will be awhile. I don't know what it is with their site, but they go to high lengths to prevent their site from being crawled. Even after using 10 second delays between each page request, I still get request timeouts every so often, which makes the whole process extremely slow. I've never used Hentai Foundry much, but they seem to have a great deal of western art, which I really want, so I'll probably make a tag archive of that next.

>>2001

No problem, it's an easy fix. Major thanks to that csv converter script though. Would have taken much longer for me to get started without it

>>2002

>support for namespace siblings / parents

That's a good idea. Some namespaces overlap, like 'origin' from MFC item db is the same as 'series' for the most part. I only used origin to keep inline with the site.

>cleaned in a centralized tag system/provider

Isn't that basically what the PTR is? Users submit tags, which can then be petitioned by other users. I've yet to see how that all works though since my client is usually in a constant state of importing files and testing tag archives.

Link:

https://mega.nz/#!KRZUgZCL!qnfS3LkKqaGyo3jbUybjPH51gNto-_RGWppQQiOsrJQ

db:hentaifoundry.db

name: Hentai Foundry

site:http://www.hentai-foundry.com/

Namespaces include creator and title of work.Keywords are considered no namespace. Contains around 200k hashes. To be honest I thought it'd be bigger, but I believe I did grab all content from every user from the site.

Sankaku chan is nearly done too. I just need to get namespace tags.

>>2056

>Hentai Foundry

Hmm, when i try to synch this archive with my db, it seems like Hydrus thinks that i have all these images and trying to synch them all. But i only have around 70k images. Other archives like gelbooru2016 or safebooru matches only few hundreds or thousands of my images.

Is there something wrong with this archive or it's problem in my db, or there's no problem at all?

Yep. There's definetly some difference between these archives. I made a new db and tried to synch it with gelbooru2016 archive. Its processing the whole archive, around 2 million files, but matched 0 because its new db without any files. The whole synchronization process takes around 1-2 minutes.

Then i made a new db again and tried to synch it with hentaifoundry archive. It matched all 207376 files and starts sync them, which much slower than gelbooru archive.

>>2072

>>2079

>>2083

>>2084

That db seems to have sha256 hashes, which means hydrus can import all its hash-tag pairs (about two million, I think!), not just the ones it can match to your local files. Syncing with that db (as opposed to a one-time import) isn't actually important, as newly imported files that are checked against it are not going to produce any new mappings, since everything will have been added already. Also, once one person syncs all that stuff to a repo, like my ptr, then no one else needs to, as there will be nothing left to add.

There isn't a way, yet, to limit the tag archive sync to a particular file domain (like local files). Would you like one?

>>2090

I don't think so. You can use page: and chapter: namespace to sort and collect inside hydrus, but I'm not really happy about the workflow. I expect I will support a multi-page, single-file .cbr-type format in future.

Have a play with a couple of chapters and the 'add tags based on filename' dialog when you import files, if you like, and let me know what you think.

>>2147

Hydrus isn't yet clever enough to do this sort of reverse lookup, so you'll have to repeat the same query with the tags checked.

Thankfully, it is clever enough to remember when it has downloaded a file from a specific url, so it won't waste bandwidth trying to download the files themselves again–it should only fetch the html page to parse the tags and then apply them to the file.

>>2175

>>2175

>To the other guy making site archives: what hardware are you using, out of curiosity?

I'm just using my cheap seedbox from https://seedboxes.cc/. Only $14 a month. I've been with them probably since 2013 now, and they're definitely reliable. Although advertised as a seedbox, they let you basically do anything on it. Of course, it's not just used for downloading images/making tag archives.. I also run h@h from sad panda on it, and also download a ton of torrents from private trackers.

20Gbps Up/Down. However, 3TB /month upload cap monthly. As for making the tag archive, it's just a simple 50 line python script that parses their index.xml page. "https://e621.net/post/index.xml?limit=1000&page={page_id}"

>short-duration EC2 c4.8xlarge instances

I actually had no idea Amazon offered this type of service. I'll have to look into it.

>(by "obscenely quick" I mean "I ripped sha1 and md5 hashes of all of e621 in about 90 minutes at 1gbit"

Yeah e621 and gelbooru can be ripped quickly. Most tag archives from big sites can be done in a few hours, with the only exception being sankaku complex. Their incredibly restrictive server makes it a huge bottleneck.

Speaking of sankaku, I had to start over. I didn't realize they used a limit of 100 instead of the usual 1000 when doing post api searches, which meant my date ranges had gotten messed up. It doesn't help that it has 10 minute time outs after 150 requests either. Makes it a huge pain