[ home / board list / faq / random / create / bans / search / manage / irc ] [ ]

/hydrus/ - Hydrus Network

Bug reports, feature requests, and other discussion for the hydrus network.

Catalog

See 8chan's new software in development (discuss) (help out)
Advertise on this site
Name
Email
Subject
Comment *
File
* = required field[▶ Show post options & limits]
Confused? See the FAQ.
Embed
(replaces files and can be used instead)
Options
Password (For file and post deletion.)

Allowed file types:jpg, jpeg, gif, png, webm, mp4, swf, pdf
Max filesize is 8 MB.
Max image dimensions are 10000 x 10000.
You may upload 5 per post.


New user? Start here ---> http://hydrusnetwork.github.io/hydrus/

File: 1445765346588.png (6.23 KB, 287x281, 287:281, BADTIME.png)

 No.1293

So I'm getting started with Hydrus. So far, I'm not ready to completely transition to it when it comes to my image management, so I don't really mind having two redundant collections since I've got HDD space to spare. As it currently stands, I've got around ~50k images I wish to import. Now, it's still at it and I've seen memory usage steadily increasing to the point it might end up crashing and I'll just have to start over.

My question is, is there something I'm missing when it comes to such large imports? It'd seem like even though a given image is already in the db, its data is kept around in memory. I could try to split things up somewhat but it'd certainly take me a while, I just want to know whether or not it'd be viable (even if extremely unwieldy) to import my entire collection whenever I just want to "sync" with my Hydrus database and expect it to skip over any files it already has included without much memory thrashing.

 No.1297

File: 1445806436349.jpg (161.88 KB, 1024x768, 4:3, d9aedfea28d1fa3dc6ff5872b5….jpg)

I am not sure why big imports are eating up memory, but I have a job in my to-do to look into it. I think there is some thumbnail data not being deleted from memory correctly. For now, if you are importing through a normal hdd import page, try to keep it to a few thousand at a time.

It probably isn't viable to manually re-import all 50k every time you want to resync. The 'is this file already in the db' check is pretty quick, but it does require reading the whole source file, which can add up.

An import folder, which regularly checks a directory you tell it to for files to import and remembers and skips the paths it has previously attempted, should do what you want, but make sure you set it up carefully. Look under file->manage import folders and click add. Since the origin folder is so large (and so will take a couple of minutes simply to parse), set the check period to 1440 minutes, which is once a day, and set every 'when a file x' action to 'leave the file alone, do not reattempt it'. Don't set anything to 'delete the file' or 'move the file', or your original collection will be changed. Every day, your client will scan your folder and try to import anything new.


 No.1303

>>1297

Alright, thank you. Setting up an import folder seemingly did the trick as indeed, Hydrus ended up running out of memory after a while. Given the relatively small increases in memory usage with every file it processed I saw, it could have been thumbnail data.




[Return][Go to top][Catalog][Post a Reply]
Delete Post [ ]
[]
[ home / board list / faq / random / create / bans / search / manage / irc ] [ ]