[ home / board list / faq / random / create / bans / search / manage / irc ] [ ]

/hydrus/ - Hydrus Network

Bug reports, feature requests, and other discussion for the hydrus network.

Catalog

The next generation of Infinity is here (discussion) (contribute)
A message from @CodeMonkeyZ, 2ch lead developer: "How Hiroyuki Nishimura will sell 4chan data"
Advertise on this site
Name
Email
Subject
Comment *
File
* = required field[▶ Show post options & limits]
Confused? See the FAQ.
Embed
(replaces files and can be used instead)
Options
Password (For file and post deletion.)

Allowed file types:jpg, jpeg, gif, png, webm, mp4, swf, pdf
Max filesize is 8 MB.
Max image dimensions are 10000 x 10000.
You may upload 5 per post.


New user? Start here ---> http://hydrusnetwork.github.io/hydrus/

File: 1444094112224.jpg (267.09 KB, 565x800, 113:160, 1cea16fa4cbf94deb21808dc2d….jpg)

 No.1180

Hi, I'm new to Hydrus.

In the past I used a downloader to grab around 100GB worth of files from gelbooru. I was wondering how easy would it be to import them into Hydrus and have tags automatically assigned to the images.

According to http://hydrusnetwork.github.io/hydrus/help/advanced.html under "tag archives", I just download the gelbooru.7z, drop it in "\db\client_archives", and let the magic happen?

Im just curious how well this will work before I spend +10 hours importing files.

Another quick question, how do I force start a subscription? I accidentally paused it and would like it to continue. Do I have to wait a day for it to start? It would be neat if there was a way to review the subscription process, so I can get an idea of how long it'll take

 No.1201

I did this recently myself.

Yes you basically download the tag archive and let the magic happen.

You need to extract gelbooru.7z so that gelbooru.db is inside "\db\client_archives".

Then it will just happen, I didn't even notice it loading, it just kind of had everything assigned already.

If you have 100GB of files though, maybe you should keep them in a seperate folder and import them sequentially, just to make sure it doesn't overload it.

The developer warns that it can be CPU intensive.


 No.1203

>>1201

Yeah I went ahead and starting importing files anyway a couple of days ago. Everything is working as should, it'll just take a couple of days to finish.

It seems I can only do around 16k files before the client runs out of memory.(Around 1.2GB of mem usage)

I still don't know how to force start subscriptions though after they've been stopped manually.(Not paused) I sent the dev an email about it but I don't think it needs to be answered now.

thanks for the response


 No.1208

File: 1444501186075.jpg (1.2 MB, 2908x1920, 727:480, e3067c250050c50cd28fb28aa7….jpg)

>>1180

>>1201

>>1203

Once you have the tag archive (gelbooru.db or whatever) in your client_archives folder, make sure you link it to a specific tag service inside your client. Go services->manage services and then select your local tag services or a tag repository, and you should see an archive synchronisation box. There you can link the tag archive, which means every file in your db will get cross-referenced to the archive, and any matching tags will be add/pended to the respective service. If you have a lot of matching files already in your db, it will do an initial sync, which can take a few minutes and may end up pending several tens of thousands of tags. The sync will then be repeated for any files you import in future.

Now I have rewritten subs, I can extend them a little easier. I want to add a 'sync now' button to the manage subs dialog, but for now you can pseudo-force it by making sure it is unpaused and going help->debug->force idle mode. Things like subscriptions only normally start when the client thinks you haven't done anything for twenty minutes or so, a bit like a screensaver. Note the sub will only start in idle mode if it is due (i.e. [the last time it checked] + [its checking period] has passed). This will be fixed with the 'sync now' button.

I am not sure why the client is crashing after large imports. Someone else mentioned similar at ~38k files. Thank you for reporting the problem. I am looking into it.


 No.1213

>>1208

>I want to add a 'sync now' button to the manage subs dialog

That's great, looking forward to that feature.

>I am not sure why the client is crashing after large imports. Someone else mentioned similar at ~38k files.

I actually think it was because I set the MB memory reserved for thumbnail/preview cache really high.(like 5k+ thumbnails) Probably a bad idea to do that when importing thousands of files. I set it both to 5 now, but just to be on the safe side I'm still limiting my imports to 10k a batch.




[Return][Go to top][Catalog][Post a Reply]
Delete Post [ ]
[]
[ home / board list / faq / random / create / bans / search / manage / irc ] [ ]