/hydrus/ - Hydrus Network

>>2332

Just installed the extract version, it doesn't give me any errors, but the process just starts and closes itself after a second.

>>2319

Options > Maintenance and Processing

will let you tell Hydrus to either perform 3 hours of maintenance on shutdown

OR

let you set "Assume the system is idle if…" to like 0-1 minutes. Hydrus will only do maintenance (which includes tag repo synchronization) if it considers your system to be idle.

Gonna ask another question, is there a problem if I'm downloading the public tag repository tags, and it's stuck on "content row 0/100,164: writing"?

It hasn't moved in a while…

>>2327

Specifically, it seems hydrus locks up when doing this (according to the db logs):

2016/03/27 14:08:34: Profiling write content_update_package

Is it at all possible to exclude files of a certain rating? Like "-system:rating for R = 0.5" or something. Or indeed, a ≠ operator.

>>2256

Your logic re new content is what I was musing about - the problem is removed tags. If there is a mapping in A that is not in B, I have no way, in the HTA, to say "this mapping should be removed if it exists". I suppose I could do an "add these" update db and a "remove these" update db, which could be one-time applied, but I'm not sure that's worth it. I'm especially not sure that running a DB that has 50,000 mappings on 50,000 files into Hydrus is going to be appreciably faster than running one that has 500,000 mappings on 50,000 files (just using numbers pulled out of my ass). Question: Does the permanent sync still take a really really long initial processing time, or does it just sit in the background?

For FA, incremental updates can and will work, though.

I like your services idea - that has potential.

Would it be possible to add a "censor this tag" option to the right-click menu you get in the selection tags menu?

>>2269

yeah that's their new thing. Make sure you edited the locale file and ran locale-gen too

After bumping up my hydrus installation a few versions last week, the thread importer is currently failing to import pictures from /furry/ threads entered with their json url, it seems to be looking for the pictures on media.8ch.net while the images for /furry/ don't seem to be hosted on that subdomain :/

>>2303

It should work if you put in the html url, like http://8ch.net/hydrus/res/173.html for this thread. I didn't even realise the json url worked previously!

The new version of the thread checker sometimes actually visits the url you put in to figure out which image domain it should be downloading from, which is absent from the json API.

>>2304

The json URLs worked just fine, they even made hydrus skip downloading items/hashes in the json which were already in the database (these would import way faster).

Now, it sadly doesn't work when putting in a html url to a thread either, now I get stuff like this (using https://8ch.net/furry/res/515077.html as an example, hydrus still seems to be assuming the images are hosted on media.8ch.net, but they're hosted on simply 8ch.net, delivering me NotFoundExceptions ):

Traceback (most recent call last):
  File "include\ClientImporting.py", line 2602, in _WorkOnFiles
    HydrusGlobals.client_controller.DoHTTP( HC.GET, file_url, report_hooks = report_hooks, temp_path = temp_path )
  File "include\ClientController.py", line 358, in DoHTTP
    def DoHTTP( self, *args, **kwargs ): return self._http.Request( *args, **kwargs )
  File "include\ClientNetworking.py", line 300, in Request
    ( response, size_of_response, response_headers, cookies ) = self._DoRequest( method, location, path, query, request_headers, body, follow_redirects = follow_redirects, report_hooks = report_hooks, temp_path = temp_path )
  File "include\ClientNetworking.py", line 249, in _DoRequest
    ( parsed_response, redirect_info, size_of_response, response_headers, cookies ) = connection.Request( method, path_and_query, request_headers, body, report_hooks = report_hooks, temp_path = temp_path )
  File "include\ClientNetworking.py", line 710, in Request
    elif response.status == 404: raise HydrusExceptions.NotFoundException( parsed_response )
NotFoundException: <html>
<head><title>404 Not Found</title></head>
<body bgcolor="white">
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.8.1</center>
</body>
</html>

>>2273

Thanks for the fix!

>>2310

>added IPFS directory parsing and download

get hype!

Thanks for the suggestion of creating some db profiles. However, I did a clean reinstal of v197 l (I also deleted the DB) and that seemed to fix it. Like you said, images now load in less than a second (probably more like 100ms).

I remember that I had a few Internet disconnects when I first synced with the repository (having some problems with my router lately, it sometimes crashes when I'm downloading with combined speeds > 100 Mbit/s). Could that have created some issues in the database (e.g., corrupt tables) that would have caused it to slow down that heavily?

Anyway, thanks again for your help. I'll probably use the OS X client quite extensively (since I won't be home at my PC all the time due to work reasons) so if you need any help with checking for bugs or optimizing stuff (like the DPI scaling) I'd be happy to help. :)

>>2287

Well, I think I spoke too soon. I have now experienced the same problem I had on my PC on OS X too. The scenario:

- Fresh Installation

- Complete sync with the repo (in the client)

- No local tags

- Import of a single image (821KB png, 68 tags in the repo)

- Vacuumed the db

- "system:everything"

This takes around 30 seconds before the loading is finished and the image is rendered. The macbook is not as powerful as my PC, but still quite the machine (2.5GHz i7, 16GB RAM, 512GB SSD). Running on the latest OS X 10.11.3. Something definitely seems fishy here.

Here the relevant part from the log I created: http://pastebin.com/6VYDvUAE

Kind of weird to me that most of the stuff shows up as taking up no time at all (0.000), but you can clearly see the one call taking over 27 seconds.

Hope you can help. :(

P.S.: Performance was fine with several hundred files before syncing to the tag repository. Since I did report that performance on PC was fine after reinstalling, I guess it may be because I didn't sync to the repository after the reinstall because I didn't find the time to do so. What I also noticed: everything seems slow in general. Opening the "manage tags" window for the image takes around a minute.

>>2287

And, since I thought it might be useful for finding the error, also a log for opening the manage tags window (which takes around a minute as I already mentioned): http://pastebin.com/tik35T4u

This one's a lot longer since there are more entries with actual time values and I didn't want to leave anything out.

>>2306

Thanks, sounds good. I'll try the update as soon as it's available. :)

Is there a way to control the order tags are listed in the autocomplete section of the 'manage tags' dialogue?

It was either because of the latest update or some setting I unwittingly changed, but there doesn't seem to be a rhyme or reason to the ordering anymore, which makes the autocomplete fairly useless at the moment

Is it just me or is the upnp broken?

>>2305

2016/03/21 20:27:09: Daemon UPnP encountered an exception:
2016/03/21 20:27:09: Exception:
2016/03/21 20:27:09: Exception

Problem while trying to add UPnP mapping:



upnpc : miniupnpc library test client. (c) 2005-2013 Thomas Bernard

Go to http://miniupnp.free.fr/ or http://miniupnp.tuxfamily.org/

for more information.

List of UPNP devices found on the network :

 desc: http://192.168.1.1:5000/rootDesc.xml

 st: urn:schemas-upnp-org:device:InternetGatewayDevice:1



Found valid IGD : http://192.168.1.1:5000/ctl/IPConn

Local LAN ip address : 192.168.1.247

ExternalIPAddress = 10.5.131.97

AddPortMapping(25510, 25510, 192.168.56.1) failed with code 718 (ConflictInMappingEntry)

GetSpecificPortMappingEntry() failed with code 714 (NoSuchEntryInArray)



Traceback (most recent call last):
  File "include\HydrusThreading.py", line 218, in run
    self._callable( self._controller )
  File "include\ClientDaemons.py", line 399, in DAEMONUPnP
    HydrusNATPunch.AddUPnPMapping( local_ip, internal_port, external_port, protocol, description, duration = duration )
  File "include\HydrusNATPunch.py", line 84, in AddUPnPMapping
    raise Exception( 'Problem while trying to add UPnP mapping:' + os.linesep * 2 + HydrusData.ToUnicode( output ) )

Hi guys, OP of the original (2014) e621 db here. I'm reripping e621 now, under my original sanitization regime - read, normalized creator: series: species: character: namespaces instead of the site's artist: copyright: species: character: native set.

Should be up in a week or two, for anyone interested.

To the other guy making site archives: what hardware are you using, out of curiosity? I've been playing with short-duration EC2 c4.8xlarge instances, and they're obscenely quick for this, but pricy.

>>2175

(by "obscenely quick" I mean "I ripped sha1 and md5 hashes of all of e621 in about 90 minutes at 1gbit". It's like renting a Bugatti to spin cookies on the front lawn of the Augusta National.)

>>2186

Ah, interesting service - will have to look into it. I need something less limited than a t2.micro, and c4.8xlarges are $1.67 an HOUR (!).

I remember the last time I did this having an insane amount of difficulty with the e621 API - part of it was that I was a much worse programmer then, but part of it was that individual API pages didn't seem to include things like tag categories and ratings. I also don't, in theory, trust their hashes - in practice they work fine, but I'm developing here with an eye towards sites like Derpibooru that optimize the image and don't update the hash in their API, so if I want anything to match on that (since as you mentioned, nobody downloads originals) I'm going to have to manually get the hashes of both the orig_ and the optimized images.

So the way I'm doing it this time is using BeautifuSoup to actually scrape every single page on the site (/post/show/{id}, iterated from 000001 to 900000 ignoring 404s), grab the tags including namespaces, grab the rating, stream the image and hash it myself, put everything in a JSON file, save it into the locally running instance of MongoDB that I keep around, repeat.

I broke up the iteration into work-blocks, which are 1000 IDs each and are served from Mongo as well. So I can fire up any number of workers, which point themselves at Mongo, grab the top block, and send the data back.

Now that I have it all in Mongo, all I'll have to do is write a quick script to iterate through it and dump it through the Hydrus Tag Archive generator.

This is definitely overkill, but it's extensible overkill - I don't need to worry about API differences, I don't need to worry about a process choking and hosing my output file, I don't need to worry about parallelism conflicts, etc.

The ultimate goal is to re-do Furaffinity, properly this time. Have to do hashing on the fly for that, too, and this method will work for sure now that their servers are less shit (thanks, IMVPost too long. Click here to view the full text.

If I synced with a downloaded db, gelbooru for example, after the sync do I still need to keep that db file or all the mappings now inside my client db?

>>2221

So if I import the tag archives the dev gives, it'll auto-tag anything with a hash it recognizes?

Also, the different databases seems to be able to be done with different installs. Which is a bit of a pain, but at least it's feasible.

>>2222

Right. The file hash is how Hydrus identifies the file. That does lead to issues with watermarked or resized files not being recognized.

There's also someone working on a script that will dump all an image's tags into a text file with the same name as the image when you save it, and Hydrus can import those tags with the image.

>>2224

And here I was thinking the tag archives/repos were just for the tags themselves. That's pretty awesome.

Thanks!

>>2263

I get that you like the portable approach, but it really has a number of problems and you really should implement a proper system to allow for installations. You're kind of bucking the standard convention of literally every OS

https://github.com/hydrusnetwork/hydrus/issues/106#issuecomment-167012174

You should take into consideration video timestamps while thinking about this.

mpv and mpc-hc save screenshots in the format: "videoname.ext [timestamp].png"

In mpv, [timestamp] is [hh:mm:ss.000]

For any screencap, I've actually gotten in the habit of adding tags in the following format:

series:archer

episode:s01e05

timestamp:00:05:01.500

I have no suggestion or request, just informing you of another use case.

I'm new to Hydrus, just testing it out for now. I downloaded the PTR sync database and about 10 different tag databases. Then imported about 1000 images. After importing the images Hydrus starts processing them and this is taking a very long time, up to 30 seconds per file. Is this normal?

I got many copies of these errors when I tried to load my inbox.

UnicodeDecodeError

'ascii' codec can't decode byte 0xc3 in position 3: ordinal not in range(128)

File "site-packages\wx-3.0-msw\wx\_core.py", line 16766, in <lambda>

File "include\HydrusController.py", line 227, in ProcessPubSub

try: self._pubsub.Process()

File "include\HydrusPubSub.py", line 127, in Process

callable( *args, **kwargs )

File "include\ClientGUICommon.py", line 4096, in SetTagsByMediaPubsub

self.SetTagsByMedia( media, force_reload = force_reload )

File "include\ClientGUICommon.py", line 4007, in SetTagsByMedia

self._RecalcStrings( tags_changed )

File "include\ClientGUICommon.py", line 3821, in _RecalcStrings

tag_string = self._GetTagString( tag )

File "include\ClientGUICommon.py", line 3763, in _GetTagString

if self._show_pending and tag in self._pending_tags_to_count: tag_string += ' (+' + HydrusData.ConvertIntToPrettyString( self._pending_tags_to_count[ tag ] ) + ')'

UnicodeDecodeError

'ascii' codec can't decode byte 0xc3 in position 3: ordinal not in range(128)

File "include\ClientGUIMedia.py", line 1948, in EventPaint

self._DrawCanvasPage( page_index, bmp )

File "include\ClientGUIMedia.py", line 1254, in _DrawCanvasPage

dc.DrawBitmap( thumbnail.GetBmp(), x, y )

File "include\ClientGUIMedia.py", line 2981, in GetBmp

upper_info_string += ', '.join( series )

I tried opening a new tab to see if maybe there were some particular images/tags causing the issue, but this new tab is loading everything fine. It's only when I switch back to the original inbox tab that the errors appear.

>>2242

Wait, I've been messing around some more. The errors are actually popping up whenever I archive an image or send it to the inbox. It has nothing to do with the image itself or its tags (it happens with an image with no tags at all, too).

>>2210

Namespace delimiters not being recursive is tricky - you could think of it as a bug, but also there's stuff like "series:the legend of zelda: ocarina of time" in which you do actually want "series" to be the namespace and the rest of it to be a full tag.

>>2211

Or less controversially, "series:2001: a space odyssey"

>>2210

Unfortunately the sub-namespaces are pretty much abstract and arbitrary. I was wondering if the delimiter could be an option or maybe two characters. The idea would be to avoid using asterisks in the search to keep the search time down, otherwise I could just put asterisks on both sides of the search term. Namespaces kind of do this already, but like you guys said it's hard to avoid recursion blowing up the search.

Ideally I'd like the search to hit on any of the nested tags individually. I picture it working like this:

set delimiter to ":."

long matches

hair:.long

hair:.long:.knee length

clothing:.shirt:.long sleeves