/hydrus/ - Hydrus Network

>>2310

>added IPFS directory parsing and download

get hype!

>>2305

2016/03/21 20:27:09: Daemon UPnP encountered an exception:
2016/03/21 20:27:09: Exception:
2016/03/21 20:27:09: Exception

Problem while trying to add UPnP mapping:



upnpc : miniupnpc library test client. (c) 2005-2013 Thomas Bernard

Go to http://miniupnp.free.fr/ or http://miniupnp.tuxfamily.org/

for more information.

List of UPNP devices found on the network :

 desc: http://192.168.1.1:5000/rootDesc.xml

 st: urn:schemas-upnp-org:device:InternetGatewayDevice:1



Found valid IGD : http://192.168.1.1:5000/ctl/IPConn

Local LAN ip address : 192.168.1.247

ExternalIPAddress = 10.5.131.97

AddPortMapping(25510, 25510, 192.168.56.1) failed with code 718 (ConflictInMappingEntry)

GetSpecificPortMappingEntry() failed with code 714 (NoSuchEntryInArray)



Traceback (most recent call last):
  File "include\HydrusThreading.py", line 218, in run
    self._callable( self._controller )
  File "include\ClientDaemons.py", line 399, in DAEMONUPnP
    HydrusNATPunch.AddUPnPMapping( local_ip, internal_port, external_port, protocol, description, duration = duration )
  File "include\HydrusNATPunch.py", line 84, in AddUPnPMapping
    raise Exception( 'Problem while trying to add UPnP mapping:' + os.linesep * 2 + HydrusData.ToUnicode( output ) )

>>2267

I set it with

localectl set-locale

which apparently just sanity-checks the arguments then writes them to /etc/locale.conf

>>2269

yeah that's their new thing. Make sure you edited the locale file and ran locale-gen too

After bumping up my hydrus installation a few versions last week, the thread importer is currently failing to import pictures from /furry/ threads entered with their json url, it seems to be looking for the pictures on media.8ch.net while the images for /furry/ don't seem to be hosted on that subdomain :/

>>2303

It should work if you put in the html url, like http://8ch.net/hydrus/res/173.html for this thread. I didn't even realise the json url worked previously!

The new version of the thread checker sometimes actually visits the url you put in to figure out which image domain it should be downloading from, which is absent from the json API.

>>2296

Hello, it seems like you might be experiencing the same problem I've had on two different machines/os's now.

I made a thread about it (ignore the stuff about DPI scaling, that's a different issue): https://8ch.net/hydrus/res/2277.html

Sounds about the same, right? I've also linked to two log extracts if you want to compare.

>>2307

Will do. I've generated pre-update profile, will do the same search post-update and post both.

By the way, do profiles have any identifying information in them? Should I email you directly, or can I just upload and post a link on the board?

>>2311

I don't think they usually do since the default install directory is c://hydrus network it avoids your name and such.

If you're paranoid about it then either read through it yourself or just email it to hydrus dev

>>2307

I've actually just gotten a huge speed increase without touching anything.

Searches now take a matter of seconds for thousands of files instead of a matter of minutes.

Did you do anything to the PTR that would cause this change?

Thanks for the suggestion of creating some db profiles. However, I did a clean reinstal of v197 l (I also deleted the DB) and that seemed to fix it. Like you said, images now load in less than a second (probably more like 100ms).

I remember that I had a few Internet disconnects when I first synced with the repository (having some problems with my router lately, it sometimes crashes when I'm downloading with combined speeds > 100 Mbit/s). Could that have created some issues in the database (e.g., corrupt tables) that would have caused it to slow down that heavily?

Anyway, thanks again for your help. I'll probably use the OS X client quite extensively (since I won't be home at my PC all the time due to work reasons) so if you need any help with checking for bugs or optimizing stuff (like the DPI scaling) I'd be happy to help. :)

>>2287

Well, I think I spoke too soon. I have now experienced the same problem I had on my PC on OS X too. The scenario:

- Fresh Installation

- Complete sync with the repo (in the client)

- No local tags

- Import of a single image (821KB png, 68 tags in the repo)

- Vacuumed the db

- "system:everything"

This takes around 30 seconds before the loading is finished and the image is rendered. The macbook is not as powerful as my PC, but still quite the machine (2.5GHz i7, 16GB RAM, 512GB SSD). Running on the latest OS X 10.11.3. Something definitely seems fishy here.

Here the relevant part from the log I created: http://pastebin.com/6VYDvUAE

Kind of weird to me that most of the stuff shows up as taking up no time at all (0.000), but you can clearly see the one call taking over 27 seconds.

Hope you can help. :(

P.S.: Performance was fine with several hundred files before syncing to the tag repository. Since I did report that performance on PC was fine after reinstalling, I guess it may be because I didn't sync to the repository after the reinstall because I didn't find the time to do so. What I also noticed: everything seems slow in general. Opening the "manage tags" window for the image takes around a minute.

>>2287

And, since I thought it might be useful for finding the error, also a log for opening the manage tags window (which takes around a minute as I already mentioned): http://pastebin.com/tik35T4u

This one's a lot longer since there are more entries with actual time values and I didn't want to leave anything out.

>>2306

Thanks, sounds good. I'll try the update as soon as it's available. :)

Hi guys, OP of the original (2014) e621 db here. I'm reripping e621 now, under my original sanitization regime - read, normalized creator: series: species: character: namespaces instead of the site's artist: copyright: species: character: native set.

Should be up in a week or two, for anyone interested.

To the other guy making site archives: what hardware are you using, out of curiosity? I've been playing with short-duration EC2 c4.8xlarge instances, and they're obscenely quick for this, but pricy.

>>2175

(by "obscenely quick" I mean "I ripped sha1 and md5 hashes of all of e621 in about 90 minutes at 1gbit". It's like renting a Bugatti to spin cookies on the front lawn of the Augusta National.)

>>2186

Ah, interesting service - will have to look into it. I need something less limited than a t2.micro, and c4.8xlarges are $1.67 an HOUR (!).

I remember the last time I did this having an insane amount of difficulty with the e621 API - part of it was that I was a much worse programmer then, but part of it was that individual API pages didn't seem to include things like tag categories and ratings. I also don't, in theory, trust their hashes - in practice they work fine, but I'm developing here with an eye towards sites like Derpibooru that optimize the image and don't update the hash in their API, so if I want anything to match on that (since as you mentioned, nobody downloads originals) I'm going to have to manually get the hashes of both the orig_ and the optimized images.

So the way I'm doing it this time is using BeautifuSoup to actually scrape every single page on the site (/post/show/{id}, iterated from 000001 to 900000 ignoring 404s), grab the tags including namespaces, grab the rating, stream the image and hash it myself, put everything in a JSON file, save it into the locally running instance of MongoDB that I keep around, repeat.

I broke up the iteration into work-blocks, which are 1000 IDs each and are served from Mongo as well. So I can fire up any number of workers, which point themselves at Mongo, grab the top block, and send the data back.

Now that I have it all in Mongo, all I'll have to do is write a quick script to iterate through it and dump it through the Hydrus Tag Archive generator.

This is definitely overkill, but it's extensible overkill - I don't need to worry about API differences, I don't need to worry about a process choking and hosing my output file, I don't need to worry about parallelism conflicts, etc.

The ultimate goal is to re-do Furaffinity, properly this time. Have to do hashing on the fly for that, too, and this method will work for sure now that their servers are less shit (thanks, IMVPost too long. Click here to view the full text.

If I synced with a downloaded db, gelbooru for example, after the sync do I still need to keep that db file or all the mappings now inside my client db?

>>1914

Ever thought about using pm2 to have the server run in the background instead of a constantly-open command window?

>>2249

Nifty, but I needed what I mentioned for existing files in Hydrus that were untagged. The way I'm doing it now is reverse image searching and AHKing the tags in, the server method would create duplicates.

The "download with tags" button doesn't work on some boorus (http://vidyart.booru.org/ as an example). I think it's because they don't have a download button and just post the original file.

>>2246

>>2261

Fixed these in 1.1.2

>>2273

Thanks for the fix!

>>2221

So if I import the tag archives the dev gives, it'll auto-tag anything with a hash it recognizes?

Also, the different databases seems to be able to be done with different installs. Which is a bit of a pain, but at least it's feasible.

>>2222

Right. The file hash is how Hydrus identifies the file. That does lead to issues with watermarked or resized files not being recognized.

There's also someone working on a script that will dump all an image's tags into a text file with the same name as the image when you save it, and Hydrus can import those tags with the image.

>>2224

And here I was thinking the tag archives/repos were just for the tags themselves. That's pretty awesome.

Thanks!

>>2263

I get that you like the portable approach, but it really has a number of problems and you really should implement a proper system to allow for installations. You're kind of bucking the standard convention of literally every OS

https://github.com/hydrusnetwork/hydrus/issues/106#issuecomment-167012174

You should take into consideration video timestamps while thinking about this.

mpv and mpc-hc save screenshots in the format: "videoname.ext [timestamp].png"

In mpv, [timestamp] is [hh:mm:ss.000]

For any screencap, I've actually gotten in the habit of adding tags in the following format:

series:archer

episode:s01e05

timestamp:00:05:01.500

I have no suggestion or request, just informing you of another use case.

>>2226

Also, did I miss a release note where the client_archives folder got deprecated in favor of the "apply archive" buttons, or am I missing the appropriate area of the UI to configure that now?

Is it at all possible to exclude files of a certain rating? Like "-system:rating for R = 0.5" or something. Or indeed, a ≠ operator.

>>2256

Your logic re new content is what I was musing about - the problem is removed tags. If there is a mapping in A that is not in B, I have no way, in the HTA, to say "this mapping should be removed if it exists". I suppose I could do an "add these" update db and a "remove these" update db, which could be one-time applied, but I'm not sure that's worth it. I'm especially not sure that running a DB that has 50,000 mappings on 50,000 files into Hydrus is going to be appreciably faster than running one that has 500,000 mappings on 50,000 files (just using numbers pulled out of my ass). Question: Does the permanent sync still take a really really long initial processing time, or does it just sit in the background?

For FA, incremental updates can and will work, though.

I like your services idea - that has potential.

I'm new to Hydrus, just testing it out for now. I downloaded the PTR sync database and about 10 different tag databases. Then imported about 1000 images. After importing the images Hydrus starts processing them and this is taking a very long time, up to 30 seconds per file. Is this normal?

I got many copies of these errors when I tried to load my inbox.

UnicodeDecodeError

'ascii' codec can't decode byte 0xc3 in position 3: ordinal not in range(128)

File "site-packages\wx-3.0-msw\wx\_core.py", line 16766, in <lambda>

File "include\HydrusController.py", line 227, in ProcessPubSub

try: self._pubsub.Process()

File "include\HydrusPubSub.py", line 127, in Process

callable( *args, **kwargs )

File "include\ClientGUICommon.py", line 4096, in SetTagsByMediaPubsub

self.SetTagsByMedia( media, force_reload = force_reload )

File "include\ClientGUICommon.py", line 4007, in SetTagsByMedia

self._RecalcStrings( tags_changed )

File "include\ClientGUICommon.py", line 3821, in _RecalcStrings

tag_string = self._GetTagString( tag )

File "include\ClientGUICommon.py", line 3763, in _GetTagString

if self._show_pending and tag in self._pending_tags_to_count: tag_string += ' (+' + HydrusData.ConvertIntToPrettyString( self._pending_tags_to_count[ tag ] ) + ')'

UnicodeDecodeError

'ascii' codec can't decode byte 0xc3 in position 3: ordinal not in range(128)

File "include\ClientGUIMedia.py", line 1948, in EventPaint

self._DrawCanvasPage( page_index, bmp )

File "include\ClientGUIMedia.py", line 1254, in _DrawCanvasPage

dc.DrawBitmap( thumbnail.GetBmp(), x, y )

File "include\ClientGUIMedia.py", line 2981, in GetBmp

upper_info_string += ', '.join( series )

I tried opening a new tab to see if maybe there were some particular images/tags causing the issue, but this new tab is loading everything fine. It's only when I switch back to the original inbox tab that the errors appear.

>>2242

Wait, I've been messing around some more. The errors are actually popping up whenever I archive an image or send it to the inbox. It has nothing to do with the image itself or its tags (it happens with an image with no tags at all, too).

>>2210

Namespace delimiters not being recursive is tricky - you could think of it as a bug, but also there's stuff like "series:the legend of zelda: ocarina of time" in which you do actually want "series" to be the namespace and the rest of it to be a full tag.

>>2211

Or less controversially, "series:2001: a space odyssey"

>>2210

Unfortunately the sub-namespaces are pretty much abstract and arbitrary. I was wondering if the delimiter could be an option or maybe two characters. The idea would be to avoid using asterisks in the search to keep the search time down, otherwise I could just put asterisks on both sides of the search term. Namespaces kind of do this already, but like you guys said it's hard to avoid recursion blowing up the search.

Ideally I'd like the search to hit on any of the nested tags individually. I picture it working like this:

set delimiter to ":."

long matches

hair:.long

hair:.long:.knee length

clothing:.shirt:.long sleeves

>>2165

ah I see now. I'm the guy who made the booru parser so this is a good find. I guess the auto-importer via import folders doesn't support file name regex or txt files (yet)?