/hydrus/ - Hydrus Network

>>678

Thank you for this report. The error here is actually wrong–it was a complicated thread initialisation problem from the db-rewrite from a couple of weeks ago. The error was not being caught correctly. This problem should be fixed in tomorrow's release.

BMP related bugs

Dragging an image file from a browser, hydrus thinks it's a bmp and then fails to find it.

Importing an actual BMP causes it to dither the hell out of the image.

>>721

This is interesting to know about browsers–it looks like the bmp is getting deleted as soon as the mouse drag and drop event is completed, and since hydrus passes import paths around a bit before actually reading the file, it is gone. I am not sure if the browser is doing this or if the OS is creating the temporary bmp file as a fallback for a raw bmp inside the drag and drop event (which hydrus does not accept).

I am not sure to what extent I can support bmp DnD operations. I will look into it and try to fix this. Thank you for reporting it.

Hydrus tries to convert .bmp to .png on import, so the dithering is probably a problem with this conversion step. Thank you for the broken example–I will try to fix it on my end.

>>727

Drag and drop from a browser to other applications does not result in a bmp, though. I think you're approaching it from a different angle, because drag and drop normally with any file- there is no conversion of the image nor does the content of the image matter for the drag and drop file operation.

That was an intact bmp. here's a dithered example

>>730

Sorry, I meant that I tried importing that bmp and got the dithered png you posted here. I was going to ask you to provide a bmp that I could test with, and then I realised that was what you posted, despite 8chan sticking a .jpg on the end for some reason.

I don't understand DnD 100%, but as far as I know, the browser is loading the DnD DataObject of an image with multiple types. It will provide, "Here it is as a bmp", which Photoshop or GIMP would be interested in; it will provide "Here it is what it links to", which is what my other browsers interpret when I do inter-browser DnD of linked images; it will provide "Here it is as a temp jpg path", which is what Explorer seems to be interpreting when I go browser->folder. When Windows handles the inter-program DnD, it tries to match what the source provides with what the destination says it can accept. Hydrus only says that it can accept filenames, and this seems to be misfiring.

When I try to do browser->hydrus, I see a path like '~/appdata/local/temp/uteohunothun.bmp'. Do you get something with a .jpg or whatever? I am using Pale Moon.

>>721

>>730

I've fixed this for tomorrow. Thank you again for reporting this, and please let me know if it does not work for you!

>>734

I had a crash myself this week while the huge 700k update was processing. I also got some RuntimeErrors and MemoryErrors while just browsing/zooming some regular files while another update was processing. The client was only using 400MB of memory, and I had a couple gigs spare overall. I suspect my graphics library is throwing a fit about generating big 'canvas bmps' (the squares of memory onto which I draw images) while the gui thread is blocking for a long time. Or perhaps some garbage collection isn't happening, or I am Destroying some objects in the incorrect order. Perhaps there is an issue with Windows timing out the new gui object if hydrus doesn't pick it up fast enough? I am really not sure. I plan to look at this next week in more detail. Please keep me updated on whether this problem gets worse or better in future, and if you notice any more symptoms or aggravating factors.

Whenever I try to refresh the account for the local file repository, I get this error and the rest of the updates fail to download. Could it be why all the images I try to view using tags from the public repo while searching using the settings "all known files" end up looking like the hydrus icon and have an unkown MIME type, preventing them from being viewed and exported?

>>738

Also here's how it looks like whenever I search a query using "all known files"

>>738

That update was the biggest yet, about 10MB for ~700,000 tags, and it is causing my parser some problems. I got an error a bit like this when I encountered the update, and then I restarted the program and it worked. This sort of thing is one reason why I am slowly converting my objects' data protocol from YAML to JSON; as well as being much faster, JSON should also be more memory efficient.

It is odd, though, because I can't figure how even 700k would push the memory limits of Windows–at most, I figure it would be about 200MB, so I guess I am calculating something wrong, or PyYAML's parser is really inefficient.

It looks to me like there is more to that error–please get it again, click 'copy traceback information', and paste it here. Alternately, it should already be written near the bottom of your client.log file, which is in install_dir/logs.

If this update does not work for you even after a fresh client restart, you can pause synchronisation for my public tag repository in services->manage services. The client will stop trying to download and parse that large update for now, and you can try again once I have fixed it.

'all known files' searches every single file your client has heard of–even those it does not have. A tag repository tells your client about files other people has tagged, and this is what is turning up in your results as in >>739 . Since they are not 'local' to your client, it can't generate a thumbnail for them, so it just shows the blank hydrus icon.

'all known files' is really just a search mode for fun or statistical research. You can see what other people have tagged without having to have the files yourself.

>>762

Thank you for the report. Unfortunately, I don't think I can do anything about that specific file. That error is from the graphics library I use, from it sperging out about a slight inconsistency in the file format. I expect the person who created that file used an unusual program to create it, and hence it wasn't 100% according to the jpeg standard. My library can't handle those sorts of errors, so it dumps out.

EDIT: Now that I look at the image more closely, I notice the grey line at the bottom. This is due to an upload that failed or a hard drive failure or similar that cut off the end of the file or zeroed it out or something. I am surprised gelbooru keeps such a broken file up!

Since these errors are fairly rare, I advise people to load the file in Photoshop/GIMP and then save again and retry the Hydrus import. Big photo-editing programs have clever custom image handling libraries that can handle and recover from errors, and then when they save, they save according to standard. The file I have attached imports to Hydrus ok. Unfortunately, it has a different hash, so it will count as a different file and you will have to add tags manually.

I'm getting a MemoryError whenever I am trying to import a 8chan thread using the thread watcher, don't recall getting this earlier.

MemoryError

Traceback (most recent call last):
  File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.ClientDownloading", line 1972, in __call__
  File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.ClientController", line 158, in DoHTTP
  File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.HydrusNetworking", line 246, in Request
  File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.HydrusNetworking", line 194, in _DoRequest
  File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.HydrusNetworking", line 489, in Request
  File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.HydrusNetworking", line 384, in _ParseResponse
  File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.HydrusSerialisable", line 21, in CreateFromNetworkString
MemoryError

>>793

Oh, and the same seems to happen to 4chan threads as well

>>792

I hadn't thought of this problem before. Thank you for reporting it. I will add an error message as you suggest.

>>793

>>794

Thank you for this. It is fixed for tomorrow's release. When I added the JSON network stuff in the big release last week, I added a handler to my network engine to catch JSON and parse it with the hydrus protocol. I forgot that the 4chan/8chan API uses JSON as well. My hydrus handler was trying to parse that and hence dumping out with a horrible error message. As is typical with this sort of thing, it was a couple of lines of code; I just forgot to think of testing it.

>>828

e621 will be fixed in the next release, on Wednesday. As >>831 says, they changed their css a little, and that broke my tag parser. You can edit the booru yourself in services->manage boorus or wait a couple of days for the update, which will do it for you.

BTW: I think the manage boorus dialog is broken for Linux, so I will be looking at that tomorrow as well.

>>838

No, this is a bug–thank you for reporting it! If it is a simple problem, which I think it is, it should be fixed in tomorrow's release.

>>1033

Thank you, I will look into it.

>>1042

Thank you–this is good timing, as I was going to work on the UPnP dialog this week. I will see about improving its error messages. Has this dialog ever worked for you?

>>1047

I think it has

I don't have such a good memory, but I was looking for a window about upnp options that I used before and couldn't find it in the main options menu

>>1061

I am sorry about this. I don't know how it is happening. Are you running from source or the executable?

Since you are on Linux, please open a terminal and go:

python
import os
repr( os.getpid() )
exit()

And tell me what you get as a result. Is it a number, or None, or does it throw an error?

You can also try:

python
import os
repr( os.getpid() )
import psutil
for p in psutil.process_iter(): repr( p )
exit()

But you might not have psutil, so don't worry if that doesn't work. If it does, have a look at the list. Is your pid from os.getpid in there?

At the moment, the 'IsAlreadyRunning' check looks through your process list, locates itself by the os.getpid call, and checks its executable and any command line arguments (p.exe() and p.cmdline() if you want to play around yourself) against all the other processes in the list. Could your os somehow be creating two copies of the process, or giving it multiple pids?

If you run from source, edit IsAlreadyRunning in HydrusData.py to just 'return False', and you will get around this problem for now. I will look at the code again when I am back at my dev machine, and if we can't fix this by next Wednesday, I'll add a switch or something to skip the test.

>>1064

I managed to reproduce this and figured it out–my Linux source was running fine, but the Linux 'executable' causes the problem. The second pid is a child of the first–I guess the Linux PyInstaller loader has to create a wrapper process or something to get Python started.

In any case, since the processes are child/parent, it is easy to detect (p.children), and it is fixed for v171. Thank you for this report!

>>1069

Thank you for this. It is related to the media viewer options I added this past week. It will be fixed in tomorrow's release.

Attempting to tag something with ":T" crashed my hydrus as soon as i hit "T"

>>1073

>>1075

Getting the tag dropdown, with all its janky event and focus behaviour, to work for all platforms and window managers is a continuing problem. I think I will add an option to have the dropdown embed just below the text entry in all cases, like it does in the manage tags dialog, and then set that to default as true for Linux.

Key event propagation in the media viewer can do with some work for all platforms. I am planning to do/continue a complete shortcut rewrite since wx's behaviour is not reliable in the cases I want to use it. That said, I will see if there is a quick solution to catch those missing key events in the Linux media viewer. Thank you for mentioning it.

>>1083

I renamed and rewrote the url downloader in v170. There was no simple way to convert old saved url pages to the new format, so they will throw errors. I don't think v170 should be able to understand the session–v169 should be the last able to do that. I meant to talk about that in the v170 release post where it was affected, but I think I forgot. I did not think that the url downloader page was very popular, so I didn't expect someone to have any of those pages saved in a session anyway, sorry!

If you don't care about those pages and what was in them, you can just load the session, ignore the errors, and save the session again. The broken pages will not be saved. If the broken pages cause any graphical mess, you can just restart the client.

If you do care about what was in the pages and want to save what was in them, you can downgrade to the last version you can get to work (I think this should be v169, but if you can get v170 to work, that's great), load the session, and select all the media in the 'url downloader' page and select 'open selection in a new page' from the thumbnails' right-click menu. This will open all the media in a new page that will survive through the update. Close the original url downloader page and save the session. When you update to v171 again, the session should load ok. If that doesn't work, let me know, and I will figure out or write a better solution. I can write a bit of manual sql to do a pseudo-update just for you, if it comes to it.

>>1086

Thank you for reporting this! It turns out that inputting ':' into the autocomplete gets chopped up and parsed into a db request for 'send me every namespace and tag ever, thanks'. This will block your gui and if the response is large enough, crash the program.

I will fix it for tomorrow's release.

>>1117

Thank you for mentioning this. I think this is probably me, so I'll check the code this week and make sure the thumb order is preserved on my end.

Hello,

on startup, after updating to latest version, approximately 9 out of 10 times i try to start the client it crashes with 23 of the following error, all the same.

RuntimeError

Failed to gain raw access to bitmap data.

  File "C:\code\Hydrus\build\client\out00-PYZ.pyz\wx._core", line 16764, in <lambda>
  File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.HydrusController", line 149, in ProcessPubSub
  File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.HydrusPubSub", line 106, in Process
  File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.ClientGUIMedia", line 2391, in WaterfallThumbnail
  File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.ClientGUIMedia", line 1182, in _FadeThumbnail
  File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.ClientGUIMedia", line 2468, in GetBmp
  File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.ClientRendering", line 428, in GetWxBitmap
  File "C:\code\Hydrus\build\client\out00-PYZ.pyz\wx._gdi", line 967, in BitmapFromBuffer

One of the times i managed to run the program, all of these cropped up on the right side as notifications. I ran all of the database operations under mainentance, and thumbnail generator gave me 13 images for which it failed to generate thumbnails. Pic related for resolutions, all of them are quite big. This worked fine before, so i'd assume something related to thumbnail max size when generating got changed during the rework of the import code, since thumbnails are one of the first things done with the pics after import.

>>1119

Thank you for this report. This is obviously serious.

Please either pause all repository sync at services->pause->repo sync or just this broken repo under services->manage services->the repo->pause sync. This will stop the error spam.

If you are comfortable navigating your file system, please go to your_hydrus_install_directory/db/client_updates. There should be one or more sub-directories with ugly names. There shouldn't be any other files. We want to find the directory that represents my tag repo. If you sync with both my repos, go to the one with more files in it. Its files' numbers end in '70343'. In this directory, find the files that begin with:

1439370343

1439470343

1439570343

And see if there is anything strange about them, for instance zero file size. My personal client has as attached. If you like, you can email me those files, or host them somewhere and post a link here.

The updates are checked for parsability when the client downloads them, whereas you are getting the error after they have been saved to your hdd, which suggests your updates have since been corrupted. A recent db update moved these files around, so did an error occur for you recently, while you were updating? Has your hard drive crashed or made odd noises recently? If you are on Windows, do you know how to run chkdsk? I can help you if you want.

No matter what has happened here, I will work on improving error recovery in cases like this. I can add a 'check updates for validity' or something button that will scan them all. And I can add better error catching for hdd errors, to pause the service for you.

>>1120

Thank you for this report. I am sorry you are having these problems. The RuntimeError is the gui library I use trying to allocate memory for some media thumbnails to put on screen and not being able to. It is a strange error. If you don't mind answering:

Which version of Windows are you using?

Do you have any window managers, like WindowBlinds, running?

If you get v172:

https://github.com/hydrusnetwork/hydrus/releases/download/v172/Hydrus.Network.172.-.Win32.-.Extract.only.zip

And extract it somewhere else, say on your desktop, will it run reliably?

If you do the same with v173:

https://github.com/hydrusnetwork/hydrus/releases/download/v173/Hydrus.Network.173.-.Win32.-.Extract.only.zip

Creating a fresh install in a new location, will it run? Will it import regular-sized files?

For the big files that failed to create thumbnails, would you mind emailing me a couple, or posting them here? I don't mind about content. If I can repeat the error on my end, I can try to figure out how to fix them. Also, did you get any errors for those thumbnails written to your client.log?

>>1123

Thank you for this update. This is odd. I was able to import and generate thumbs for all those big files with no problems. I looked up the 'broken data stream' error, and it seems to usually be a mismatch between dll-type objects. It may be that your install_dir/PIL._imaging.pyd file got corrupted somehow, or there is some odd-versioned jpeg dll being loaded by accident. I am not sure. If a fresh install works, then that seems a good solution.

I know nothing about junction links, so I can't talk with any expertise about that, but 3-6 seconds seems a very long time just for some quick tags. When the client does db work, it actually usually accesses client.db-shm and client.db-wal, which will appear and disappear when client.exe runs. As I understand it, once the WAL file gets a whole bunch of transactions, it'll compile them into a neater package to actually write to client.db. There is some low-level file locking going on here, so perhaps straddling that EXCLUSIVE lock across different drives and a link is causing excess latency? DB slowdown is often because of latency rather than throughput, which is often an issue in RAID deployments, for instance. The low-level speedup I did a while ago made the WAL less connected to the main .db file, so this may be it. If you haven't tried running the client.db on your hdd, without the link, give it a try. I would not recommend trying to create a link for the shm or wal files unless you have a backup ready, as these files are pretty volatile, and screwing around with them can easily corrupt your db!

The size of your db shouldn't slow things down–my hdd laptop runs fine with a 1.9GB, 250,000 file db–but running database->maintenance->vacuum is usually good to try if you don't have it set up to automatically run.

>>1124

Here you go:

http://www.mediafire.com/download/r68v80zwg81w8p5/updates.zip

I put the neighbours in, just in case they are a problem as well.

Thanks for this. As you say, this is a good error to fix–I'll add some better recovery options for this sort of problem.

Some users have said hydrus works well with TrueCrypt if you haven't tried that.

>>1127

Hello there,

the problem reappeared in the new installation as well, so i did whatever i could to fix it, database mainentance, using the sqlite console to check integrity and clone it, without using the different hdd link this time, so everything is stored at one place, but it still appears.

About 9 out of 10 times, the client just simply freezes after loading all the tabs of the last session, but before loading the sidebar or the pictures (pic related), and the standard Windows "application has stopped working" window appears, stating that the cause of the issue was APPCRASH, and the conflicting module is "wxmsw30u_core_vc90.dll", version 3.0.1.0. In the client log varying amounts of the "Failed to access raw bitmap data" errors spawn up.

The one time it starts, it either spawns a LONG row of these errors as notifications (this makes me think error handling on startup while loading previous session might be too heavy load so it crashes), or, strangely, no errors at all, which leaves me puzzled. But, the database seems to have become corrupt instead, not letting me apply the latest update from the public repo, with this error:

Database Error!
constraint failed
Stack Trace (most recent call last):

  File "C:\code\Hydrus\build\client\out00-PYZ.pyz\threading", line 783, in __bootstrap
  File "C:\code\Hydrus\build\client\out00-PYZ.pyz\threading", line 810, in __bootstrap_inner
  File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.HydrusThreading", line 127, in run
  File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.ClientDaemons", line 285, in DAEMONSynchroniseRepositories
  File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.ClientData", line 1687, in Sync
  File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.HydrusController", line 224, in WriteSynchronous
  File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.HydrusController", line 59, in _Write
  File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.HydrusDB", line 320, in Write
  File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.HydrusData", line 1263, in GetResult

Traceback (most recent call last):
  File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.HydrusDB", line 181, in _ProcessJob
  File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.ClientDB", line 6350, in _Write
  File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.ClientDB", line 4192, in _ProcessContentUpdatePackage
  File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.ClientDB", line 4446, in _ProcessContentUpdates
  File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.ClientDB", line 3149, in _GetNamespaceIdTagId
IntegrityError: constraint failed

While trying to replicate the problem on a fresh installation, with a clean database, no such errors have appeared, but i also have a lot of tagged pictures that i would not like to lose, so i thought, can i run a local server, force the old client to export all the tags in it's local db to the local server, start a fresh installation, import all of the old pictures and then download tags for them from the server? It all seems to be tied to the database somehow, but i lack the means to find the exact issue with it, as the console's integrity check and the client's internal thorough file integrity check report no errors or corrupted files, yet it keeps crashing.

Also while clearing all those tabs to force restart with a clear session, file count from the last closed tab keeps being displayed on the status bar in lower left corner, just a small thing :)

Thanks for your help, and cheers for your hard work yet again!

P.S.: Short article about Windows file links, including junctions, if you're interested. http://devtidbits.com/2009/09/07/windows-file-junctions-symbolic-links-and-hard-links/

>>1131

Maybe the session load is creating a lot more thumbs than it should, so the 'access raw bitmap data' error is actually an 'out of memory' error. If you open up task manager (Ctrl+Shift+Esc) when the client loads, does it use significantly more than ~160MB of memory on boot?

I'll add in a little delay to my error catcher, in case the spam is overloading something. Incidences of spam can do with a bottleneck anyway.

That db error looks serious. It means the tags_fts4 and tags tables are no longer synchronised, which shouldn't normally be possible. Maybe a transaction got committed despite only being half done, or a tag got deleted somehow, or a sector got corrupted. If the sqlite console can't fix it, I think recreating from scratch is a good idea. You can copy your local tags from one client to another really easily by using hydrus tag archives. Go services->review services->local tags->perform a service-wide operation. There are a couple of buttons that will let you export all your service's tags to an external file, and import them again in the same dialog on your new install.

>>1144

Ah, it looks like the guy who wrote send2trash, the library I use for recycle bin stuff, merged the Python2 and Python3 versions together and missed that Python2's str() can't do encoding on creation!

I'm not sure how I'll fix this. I'll probably have to encode unusual paths before I pass the path along.

Did you happen to have unicode characters, or japanese or something, in the filename that was to be recycled?

I think I found a small one, Linux Mint x64

I can not archive/delete or use a custom filter on a group of files that is being displayed as a single collection thumbnail. The context menu entries are present, but have no effect and offer no error message upon selection (A custom filter can still be set up, but not sucessfully applied afterwards).

It works fine on a selection of multiple single files, just as long as every file has its own thumbnail.

Opening a given collection in a new page, disabling collection settings and then selecting all files to filter works well enough as a workaround, but being able to apply filters without the extra steps would save some time

Not a bug, but I keep getting these database full errors.

I usually just ignore them.

Should I be concerned?

Is it normal?

>>1182

I was actually having the same issue and discovered that it was an issue with when the database was vacuuming, which it automatically does every so often (I believe you can change this interval or even disable it if you just want to ignore it, but it might end up slowing hydrus down in the long run).

Hydrus uses SQLite, which apparently requires "as much as twice the size of the original database file" [ https://sqlite.org/lang_vacuum.html ] (specifically in your temporary directory).

This just means that your hydrus database is taking up more than half of the space available in your temporary directory (likely /tmp if your on linux). This isn't necessarily how much disk space you actually have left on your hard drive, since many (all?) linux distros mount your RAM through tmpfs to the /tmp directory instead.

The way I got around this was inspired by https://stackoverflow.com/questions/23249843/sqlite3-vacuum-database-or-disk-is-full . I edited my /usr/bin/hydrus-client to use a custom directory I made on my hard drive as it's "temporary" directory by adding "export TMPDIR=<new directory>" before "exec python2 -OO /opt/hydrus/client.pyw "$@"" (the second line may be different from what you have, so just replace it with whatever your hydrus-client script has) and so from then I could vacuum without errors.

#!/bin/sh
export TMPDIR=<new directory>
exec python2 -OO /opt/hydrus/client.pyw "$@"

You will probably have to open whatever editor you use as root (using sudo) to edit this, though. I'm not sure how to fix this on different OS's, but it's likely a similar process. This will probably be overwritten in the future on any new hydrus updates, also, so you will have to re-edit it every time until the dev adds an official fix for this.

>>1186

Your version of python pillow is too new, downgrade to before 3.0, or wait for hydrus_dev to fix compatibility.

>>1182

>>1183

>>1185

Thank you for this report and explanation, as I would not have figured it out. I will add a note to my help files about this and try to catch and recover from that explicit error.

>>1186

>>1193

And thank you for this report and explanation. I have updated all my dev machines to Pillow 3.0.0 and fixed the obselete calls. This will be fixed for v177.

>>1207

Just a heads up, OpenCV has a 3.0 stable as well, which is also currently incompatible. Hydrus will boot but is unable to download images from repositories.

>>1216

I have been regularly checking for a Windows OpenCV package here:

http://www.lfd.uci.edu/~gohlke/pythonlibs/

And the new 3.0.0 doesn't have a 2.7 release. I assumed it was incompatible. Does someone else make a simple package for this? I think I remember trying to compile this myself before on Windows and it was just pages and pages of compiler errors and lonely forum posts from 2008.

Also, do you know if the apt-get/homebrew calls for this different? It looks like homebrew supports having 2.4.12 and 3.0.0 simultaneously, so maybe the python package is also cv3 instead of cv2.

Did you get any errors when you tried to import images? Until I can figure out how to update for all platforms, I'm happy to support both versions as much as I can.

>>1219

I hate regex. What you have there gives 'unbalanced parenthesis' for me, but after some rejiggering I got an error about the look-behind that reminded me you can't have a look-behind with an uncertain number of chars. So, something like [0-9][0-9][0-9][0-9] would work instead of [0-9]+, if you are always looking at four numbers. I will get the client to pass up regex compile errors when you click ok/add. I think at the moment they are silently ignored.

>>1226

How are you importing it? I think there is usually supposed to be a popup message with some progress gauges when you do this sort of stuff, but perhaps your CPU is getting hammered so hard that it doesn't have time to pop up. In general, if you are unsure if hydrus is just busy or hung, you can open task manager (Ctrl+Shift+Esc for Windows) and look for high CPU load on client.exe. Depending on how you are adding it, it could take a few minutes or a few hours, I am not sure. I haven't done it myself, and the hashes are different to the other tag archives, so it may be that all of its data is importable.

>>1227

I run gentoo and didn't realize there isn't a py2.7 release for windows of OpenCV. Gentoo lets me compile OpenCV targeted for python2.7 and it does work; the module is still named "cv2".

I don't know if the apt-get/homebrew calls are different, I run Debian unstable as well and it doesn't look like the package maintainer is pushing OpenCV 3.0 yet.

I tried my luck at compiling OpenCV 3.0 on windows for python 2.7 (win32) and seemed to have succeeded. I have default build config options set, I don't know if hydrus requires or wants OpenCV to use CUDA and OpenCL (probably not) but they're baked in at the moment. I'll provide a link to my build here, which is literally just the dlls and pyd zipped together; I have no idea how to make a whl from the build results:

https://my.mixtape.moe/wkasyz.zip

This will probably throw an error requiring nvcuda.dll or something like that on systems that don't have nVidia drivers installed. I might attempt to setup some sort of CI for OpenCV or at least provide a config.

I probably should have posted the traceback that occurs when attempting to download files:

http://pastebin.com/NtF5gy9z

A similar (fugly) traceback is thrown when importing:

http://pastebin.com/R76TmRq4

(Pastebinned because body was too long)

>>1243

Looks like hydrus_dev missed an obsolete call, I didn't realize Pillow obsoleted another call either.

If you *really* want you can find and replace the call of "fromstring" to "frombytes" on line 233 in include/HydrusImageHandling.py and it might work, or wait for next release again.

>>1243

>>1245

Whoops! My test routine should have caught this, not sure why it didn't. :/

I'll fix it next week.

>>1244

Thanks for sorting this out! I will have a look at this this week.

>>1248

Thank you for reporting this. I think this is a new change on their end, as I think I remember using gelbooru for a different test just the other day.

I might as well send the referrer header for all the gallery downloads, just in case anyone else adds this sort of thing. I will make sure to get this done this week.

>>1251

Thank you for this report. On my Linux (Ubuntu 14.04), both modes work as expected. Do you have any relevant-looking tracebacks in your client.log? It should be in install_dir/logs.

What sort of query are you using for your export? Several tags? I notice now I look that most of the system predicates don't work properly, so I'll make sure to fix that.

Are the search domains set to 'local files' and 'all known tags'?

And what export phrase are you using? Might several files end up with the same filename?

I am fairly sure that regular mode isn't able to run any delete code. Do you have multiple clients pointed at the same directory, or maybe an import folder set up on the same location, or another program that could be doing the deleting?

Having multiple queries is a good idea, thank you. I had planned to allow multiple export folders for the same destination, but now I think about it, that won't work with synchronise, as the competing export folders will delete each other's results. I will change my to-do list.

Hydrus v177, pillow v3.0.0

This file won't play (hydrus jumps to the first file in the list) and I get an error when selecting it:

AttributeError
'NoneType' object has no attribute 'dirty'
  File "/opt/hydrus/include/ClientGUICanvas.py", line 266, in EventResize
    self._video_container = ClientRendering.RasterContainerVideo( self._media, ( my_width, my_height ) )
  File "/opt/hydrus/include/ClientRendering.py", line 186, in __init__
    self._renderer = HydrusVideoHandling.GIFRenderer( path, num_frames, target_resolution )
  File "/opt/hydrus/include/HydrusVideoHandling.py", line 428, in __init__
    self._InitialisePIL()
  File "/opt/hydrus/include/HydrusVideoHandling.py", line 514, in _InitialisePIL
    self._pil_dirty = self._pil_image.palette.dirty

>>1253

Thank you for this report. It will be fixed in tomorrow's release. Interestingly, this file only caused problems in non-Windows, so I suppose Pillow v3.0.0 is slightly different between platforms.

>>1275

If both services are saying something to do with tags, then something has gone wrong. I am not sure why a 'regular' export folder might delete, but perhaps some ids are getting mixed up and it is truly a synchronised that is finding zero results and deleting everything else.

If you don't mind, please close your client, go into your install_dir/db directory and run sqlite3. This should give you a sqlite prompt. Then go:

.open client.db

.once results.txt

SELECT * FROM json_dumps_named WHERE dump_type = 16;

.quit

Which will write your export folder info into 'results.txt'. Then email me that file, or post it here if there is no personal info. I will look for anything out of place.

>>1287

That believes it is a synchronise folder, which suggests your client is not correctly storing and retrieving the behind-the-scenes information in the dropdown where you select regular/synchronise. 6c6f63616c2066696c6573 is also 'local files', which is somehow presenting as 'local tags' for probably a similar reason–some ids are getting swapped around inside the gui library. Something similar to this happened for another Linux user, where a standard behind-the-scenes data call in wx (my gui library) just failed for certain controls, so I had to rewrite them.

Does it show 'synchronise' in the normal manage export folders dialog where all your export folders are listed?

Inside the edit export folder dialog, does the regular/synchronise dropdown list them with regular in the first position and sync in the second?

If you click on the button on the left, where it says 'local files/tags', what do you get listed in the menu?

Do your normal search pages work ok? Have you ever had 'local files' or 'all known tags' swap out for a different search domain?

Also, which flavour of Linux are you using, and do you know which window manager you have?

>>1275

>>1287

I just realised we might be talking about a different thing on the 'local files' and 'all known tags' issue. I mean do the buttons look like my picture here? The one on the left says 'local files' and the one on the right says 'all known tags' i.e. the export folder will search your local files for the 'bg' tag on any tag service.

I assume you get 'local tags' and 'all known tags' in the menu if you click the button on the right. If you don't have any tag repositories added, then 'local tags' and 'all known tags' are essentially the same thing.

If your buttons are like that, then they are working fine, but it is a new mystery why your export folder is not generating results for that query (or at least after the first run). I presume a normal query for 'bg' gives you a bunch of files, right?

>>1309

Ah, shit, I read that json dump wrong. Sorry, that is indeed a regular folder, not a sync. Since you also get 'regular' everywhere in the right places in the gui, that crosses the gui off the list as the possible problem.

I am completely not sure what is going on here, then. If you are still keen to figure this out, I will include some special debug code in tomorrow's release that will throw up a lot of debug popup messages while the export folder runs, and we'll see if any of the search results or destination paths are coming out wrong. I'll add an entry to help->debug.

>>1309

>>1315

I just put the release out. If you are interested, turn on help->debug->special debug mode while the export folder runs. It will dump a lot of messages to the popup manager and also silently print some longer stuff to the log as well. Close the client and then check your client.log file for the export log stuff. There might be several thousand lines. Email me that extract (I suggest you not pastebin it, as the file hashes are private information), and I will have a look.

I get this when I try to sync with the tag repo

UnicodeEncodeError
'ascii' codec can't encode character u'\xa0' in position 100: ordinal not in range(128)
Traceback (most recent call last):
  File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.ClientData", line 2324, in Sync
UnicodeEncodeError: 'ascii' codec can't encode character u'\xa0' in position 100: ordinal not in range(128)

>>1327

Funnily enough, the splash screen was stay_on_top until a couple of weeks ago. I turned that off because it was causing different problems.

When you say it appears behind every window, do you mean it actually initialises in the background? That doesn't happen for me on Win7/8. Just for interest, which version are you using? Do you have a window manager or any other sorts of gui-addons that might mess around with that stuff?

Also, when you have the password dialog waiting for you, is the hydrus splash screen listed in the Alt-Tab list? I see it in there (so I can bring it up top that way if I need to), but I do not see a taskbar tab until the main gui launches.

I have told the splash screen to explicitly try to bring itself to the front as soon as it opens. When you update to v180, see if this is fixed, and let me know if it isn't.

>>1328

When the mime goes through to the predicate list at the top-left, does it say system:mime is image? Which os are you using?

>>1331

I updated some OpenCV code this week, and that xcb_xlib… error gives some OpenCV errors when I search it, like this one:

http://stackoverflow.com/questions/11513564/opencv-multithreading-error-using-qt

So maybe that's it? I don't have a debian machine to test with–did previous versions ever work ok for you? I assume you are running from source.

If you open up a terminal and go:

python
import cv2
cv2.__version__ (that's two underscores both before and after)
exit()

What does it out put as the version? 2.4.x or so, or 3.0.0?

>>1336

Thank you for this report. Are you running on a non-english os, by any chance? Or have you given your repository an unusual name?

Still, even if you have odd characters (that's a non-breaking space it is having trouble encoding) I want my code to be able to deal with it. I will make sure to have a look at this this week and fix whatever is going on.

>>1345

I have rewritten my unicode handling for next week. There were several bad bugs. Please hit services->pause->repository sync for now and try again in v180.

>>1347

About the background, I mean that if you have a other programs open and then launch hydrus, does the splash appear in front of those windows, or is it quietly created behind them? Windows usually puts new programs on top of the window stack by default, but I would be interested if Win10 is handling my splash screen any different.

The raise call I added for next week will explicitly move my splash to the front of the stack.

The mime bug is actually something I accidentally introduced last week, as per >>1326 , that will be fixed next Wednesday. I screwed up the add/remove code for predicates that have internal variables. Your predicate is searching correct, but when you try to remove it, the list of active search predicates misidentifies the action as 'add new predicate' and so is incorrectly launching the dialog to give its variables some values. I'm sorry for the confusion.

For this week, if you reenter the exact same information as the existing predicate, it will be removed, otherwise it will add a new predicate.

I'm not sure if this is intended behavior or not, but I've noticed that upon setting up a subscription, for some *booru sites (e621 most specifically) Hydrus will dutifully download and check the first X (where X is the initial file limit) files on its initial sync and any new files added after that. However, it doesn't seem to bother checking any files before those in subsequent sync operations, even if there's no such limit.

With a more specific example: I've left the limit for the initial sync at 200. Since my last synchronization operation, I've added 18 images to my favorites on e621. Now my file count is at 218 after the next sync ends. However I've got ~2000 favorites, the rest seems to never get checked.

I'm wondering if it's a peculiarity of the way files are sorted by e621 or if I'm doing something wrong on my end.

>>1351

>>1353

>>1354

>>1355

>>1356

>>1358

This was reported through github and email as well! I think I have it fixed for tomorrow. The precompiled binaries have cv2 in them, so if you don't have it from the terminal, that is probably OK. I thought this was PIL or OpenCV to begin with, but I currently think it is caused by my new 'is the mouse idle' code accessing the current mouse coordinates from the wrong place, causing X11 to sperg out. I don't get the crash on my Linux laptop anymore, although I wasn't getting it much before, so I am still uncertain. I'm sorry for the inconvenience. Please let me know if any of you still get this crash in v180.

>>1352

How are you getting your favourites in e621 through hydrus? Can you do that by tag, or do you have a custom e621 search defined?

Subscriptions always get 'newer' files. The subscription gallery parser grabs new thumbnail-urls until it sees one it has seen in a previous run. So if you have a gallery like this:

TUVWXYZ

With initial limit set to 3, the subscription will initially get T, U and V and then stop looking. On a subsequent check, it might get:

PQRST

And then it will add P, Q, R, and S to its cache of TUV and stop looking, because it already saw T once. The sub will never get WXYZ.

If you want to get the 1800-odd files your sub missed, you could recreate your subscription entirely or just reset its url cache (which you can do in services->manage subs) and set its initial file limit to 0 (for infinite) or 3000 or whatever. It will parse through the 35 or so pages and slowly download everything. That might be a little scarily large to leave to run by itself, so I'd advise you instead leave your current sub as it is and instead just do a one-time gallery download page with file limit of 0 and watch it work manually. Once it has built up the whole list of files, you can periodically pause the file downloader to stop that rabbit from melting.

I'm getting this when adding pixiv to the default tag import options:

UnboundLocalError
local variable 'namespaces' referenced before assignment
  File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.ClientGUIDialogsManage", line 3714, in EventAdd
  File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.ClientGUIDialogs", line 778, in __init__
  File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.ClientDefaults", line 341, in GetDefaultNamespacesAndSearchValue

pixiv doesn't show up in the list after this traceback is thrown, it only seems to be pixiv as well I can add deviant art, tumblr, etc.

Upgraded to v180 and now I get this whenever I make a search.

UnicodeDecodeError
'ascii' codec can't decode byte 0xa0 in position 2: ordinal not in range(128)
  File "C:\code\Hydrus\build\client\out00-PYZ.pyz\wx._core", line 16764, in <lambda>
  File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.HydrusController", line 172, in ProcessPubSub
  File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.HydrusPubSub", line 106, in Process
  File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.ClientGUIMedia", line 990, in SetNumQueryResults
  File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.ClientGUIMedia", line 631, in _PublishSelectionChange
  File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.ClientGUIMedia", line 971, in _GetPrettyStatus

>>1363

Does that artist have a lot of tag siblings? My autocomplete and taglist collapses sibling counts together, so if your database would otherwise have:

creator:lm (20)

creator:lm (legoman) (10)

creator:legoman (100)

You will see creator:legoman (130), when the true answer should probably be 105 or something. Trying to get an accurate count in this case for the autocomplete takes a lot of CPU, so there isn't a good solution. Now I think of it though, I might be able to make the selection tags more accurate.

You can see all the siblings separate if you open the manage tags dialog.

Let me know if this isn't a siblings issue, and we can look deeper.

>>1365

Thank you for this report. It is fixed for next week. The Hentai Foundry 'parent' entry had the same problem propagating down to its 'searching by artist/tag' children.

>>1372

I am sorry for this problem. All my work on unicode was supposed to fix this stuff!

It looks like your OS is converting a large number to something my code is not expecting. When you see a big number on your computer, what do you see? Something like "123,456"? A comma every three places, or something else? Which language does your OS use? And what OS are you using? Is that Windows 10?

You can revert to v179 this week–there are no db differences. I have changed the call to your OS to try to fetch it in unicode, which may fix your problem. If we don't figure it out for certain this week, please let me know how v181 works for you.

>>1373

I noticed in testing your bug that page up/down didn't work, and was amazed that I hadn't added it. I am almost certain that once worked, which just goes to show how reliable memory is.

I had this sort of graphical glitch when I first wrote the thumbnail panel. It was something to do with the scrolling event that asks to render a new row of thumbs going before/after the scroll actually occured, so the 'current' top/bottom row was not getting calculated right. The exact scroll timings across all platforms ended up being such a pain in the ass to get right that I rewrote everything to work off the 'paint' event that your OS sends when it wants to redraw the window. The paint window shouldn't be vulnerable to this error, so something odd is going on. I am guessing something like your window height is an exact multiple of the thumb height, so my logic is getting +/-1.

I had a look at my code, but nothing stood out at first. I simplified things, and even optimised a little, and found a new glitch that I can reproduce (a big white box where some thumbs should be) that may be a different version of what you are seeing. There's obviously something bad in here, so I'll keep working on it. If I forget to update you on how I get on, please let me know how v181 works for you.

>>1379

I would guess large numbers use commas as I have Swedish keyboard layout, date, time and number formatting. The rest of the os is in English windows 10.

>>1379

>Let me know if this isn't a siblings issue, and we can look deeper.

I just found out it was because 362 of the 428 images were part of a collection. I guess it did it automatically.

How do I actually view these images within the collection, and is there a way to remove the collection?

Not really a bug, but the interface works kind of bad in a small screen.

I'm using a 1336x768 screen.

Things like these keep happening.

when i search for a tag (while trying to add it to an image), it gets this error.

it doesn't always do this, sometimes it works fine

UnicodeDecodeError

'ascii' codec can't decode byte 0xa0 in position 1: ordinal not in range(128)

File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.ClientGUICommon", line 478, in TIMEREventLag

File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.ClientGUICommon", line 536, in _UpdateList

File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.ClientGUICommon", line 2913, in SetPredicates

File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.ClientData", line 1310, in GetUnicode

>>1375

>>1376

On your search page, the autocomplete dropdown has some buttons underneath. Does your 'files' one says 'all known files' or 'local files' or something else?

Also, do you sync with my public tag repository? Are these files tagged k-on on your 'local tags' service?

'All known files' searches for absolutely every file your client has heard of, no matter how little it knows about them. I think that is what you are seeing–typically, you get these hydrus icons on files that your client has incomplete knowledge of. You might not have the file, but you know that it is tagged it with k-on or whatever. (deleted files would be included in this)

New pages default to 'local files', which is probably why your new page is only showing files the client actually has.

It might also be that the delete code is misfiring. If you are getting 'unknown' files in a 'local files' search, something is very wrong.

I will fix the copy->path event to restrict to local files only, thank you for pointing it out.

>>1386

At the top left, where the dropdown says collect by creator-series, you can set it.

The media viewer will scroll through collections as if they were 'flat', so just double-click any collection and press page down or whatever.

I am not very happy with how collections work right now, so let me know if you have any ideas for improvements.

>>1387

I am sorry about this. Another user mentioned this recently as well. I have a priority job to add scrollbars or and better sizers to all my dialogs. Please keep me updated as I roll this stuff out if anything doesn't work for you.

>>1393

>>1394

Thank you, yeah, this seems to be the same problem. I expect the problem is occurring for tags that have a count greater than 1000.

In the big unicode conversion last week, I missed this one line where I send big integers off to the OS to be converted to whatever your language settings would prefer. For English, 123456 becomes 123,456 with a standard comma, but I presume your language uses a slightly different comma that is in unicode, and because I missed the single line, this complicated character doesn't get incorporated back into my program correctly.

I think I have it fixed for v181. If you want, you can revert back to v179 for now. Thank you for the report!

>>1406

Thank you for reporting this. For now, restarting your client should fix it. Let me know if it doesn't.

It looks like parent and sibling managers were not being notified of the changes when new relationships were were added via repository processing. This should be fixed for tomorrow.

Trying to display a 8340x10470 spikes my memory usage to around 1gb. Which then prompt memory errors. where did I go wrong?

Not sure how I can go about helping to debug the issues im having, but, once I hit 100k+ images, my installation has gotten even slower, massively so. I added a very large collection to my local hydrus, and it got very laggy, and very slow. Often it will crash when trying to type in tags, thats the biggest area where it hangs, and also the most important.

Perhaps an option to type in tags then hit search,instead of it automatically searching, which seems to be what is causing the hang, when its trying to load up info of 100k files.

Thats mostly where it hangs up, when trying to load the files which typing in or searching tags seems to start trying to do, this is where it gets the worst.

I have also moved it from my ssd to hdd, i had SOME better speed on the ssd i think, but i don't think thats the only problem.

I have in the past, noticed some speedups when changing the process priority to high

And yes, if you are wondering, this is only one series of images, all Rozen Maiden, I have not added any other image collections to this, so its not as simple as removing unrelated images or splitting different types of images into different installs

Is there any other info I can provide to help pinpoint how you can fix these hangups?

>>1411

I don't think you did anything wrong. My media drawing code can't handle big images very well yet. At the moment, it tries to render the whole image at 100% as soon as you zoom in, which means in your case a 8340x10470x24bit depth = ~260MB bmp. There can be several copies of an image held to cache certain processing steps and to buffer from one format to another, which is probably what is smashing your memory.

I plan to rewrite my canvas code to draw lots of 400x400 squares or whatever to tile the image, streaming in zoomed-in segments only when they are on screen, which should fix your problem. For now, please give your big images a local tag like 'high res, leave for now' and archive it, and come back to it later when Hydrus can manage it.

>>1414

The biggest slowdowns I have noticed are intense HDD activity (like a defragger running or another hydrus client doing processing) and a fragged hard drive. If you recently added those files, your db and db file cache is probably a bit fragged, so run your defragger and a database->maintenance->vacuum inside the client to internally defrag the db.

Beyond that, I agree we are starting to push my code with the current state of the public tag repo. Do you sync with it on that 100,000 client? The combination of six-figure files and 20 million tags blats my laptop, especially in making large-result autocomplete queries.

A big project that I am pushing nearer to the front of the queue is streaming autocomplete results, with the aim of always getting some kind of response in preferably 0.5s or so. The list will populate in the background, giving the gui some cpu time to unhang so you can select the (usually popular) result you want while the smaller results load out of view. Or it might take a different form, depending on the exact search logic, maybe loading the results quick but streaming the cpu-heavy file counts in or something.

To contribute, I am always interested in profiles of slow commands in real-world environments. Hit help->debug->db profile mode and then do something that is slow. Any db requests will be profiled and a big timing table will be written to client.log. Cut the appropriate text and email/pastebin it to me, and I'll have some great hints on what exactly is going slow for you.

>>1417

Running that vacuum did wonders, immediately noticeable improvements

Before I had lag no matter what tag i was trying to type in, whole hydrus program would stop completely for a moment. Now, I type in, no freeze of program, split second later the files start to load

I have it synced with the public tag repo.

If i see it lag up in any other instances I will use the db profile mode

>>1372

Version 181, same problem as before.

UnicodeDecodeError
'ascii' codec can't decode byte 0xa0 in position 0: ordinal not in range(128)
  File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.ClientGUICommon", line 584, in EventMenu
  File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.ClientGUICommon", line 536, in _UpdateList
  File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.ClientGUICommon", line 2930, in SetPredicates
  File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.ClientData", line 1338, in GetUnicode
  File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.HydrusData", line 172, in ConvertIntToPrettyString
  File "C:\code\Hydrus\build\client\out00-PYZ.pyz\locale", line 196, in format
  File "C:\code\Hydrus\build\client\out00-PYZ.pyz\locale", line 217, in _format
  File "C:\code\Hydrus\build\client\out00-PYZ.pyz\locale", line 166, in _group

>>1418

That's great. It might also be that running a vacuum simply puts all your db pages through Window's file cache, so your access is sped up because hdd latency is temporarily eliminated, so let me know if things are running slow again 48 hours later.

I think I have hydrus set to vacuum every five days by default, but I am completely unsure what I should really be aiming for. Maybe I should queue it after every big insert. Have you changed yours? It should be under options->maintenance and processing.

>>1419

It turns out my preferred fix only pushed the error into the library, which it turns out can't handle unicode conversion of weird characters either!

I put out a hotfix last night, here are the links:

https://github.com/hydrusnetwork/hydrus/releases/download/v181/Hydrus.Network.181.Unicode.Hotfix.-.Win32.-.Extract.only.zip

https://github.com/hydrusnetwork/hydrus/releases/download/v181/Hydrus.Network.181.Unicode.Hotfix.-.Win32.-.Installer.exe

I have only heard about this bug from Windows users, so I only made a Windows hotfix, but if you aren't, let me know and I'll put a Linux/OS X version out.

If you are already using the hotfix and still getting this problem, let me know!

>>1427

Tried the 181 hotfix. It works as it should except syncing the tag repo.

UnicodeEncodeError
'ascii' codec can't encode character u'\xa0' in position 100: ordinal not in range(128)
Traceback (most recent call last):
  File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.ClientData", line 2344, in Sync
UnicodeEncodeError: 'ascii' codec can't encode character u'\xa0' in position 100: ordinal not in range(128)

>>1431

Thank you for this. I will rewrite how things get printed to the log for next week to eliminate these new errors. Please pause repo sync for now by going services->pause->repository sync.

>>1435

Oh, and here I thought I was the only one with a big collection. Glad to see otherwise.

I just tried messing with the speed and memory options, upped the memory space for all functions a little, added another 200 ms to autocomplete long wait, and im noticing the difference I think. Thanks, that seems to help.

>>1438

That sounds optimal, and awesome. I had hydrus on my ssd before, but the repo is too big to sit on the ssd with the operating system, unless i removed the whole linux partition (i have a windows/linux dualboot)

How did you do that that would also be awesome if it were supported, wink wink nudge nudge hydrus dev

>>1455

>>1466

Although I'm not in 'new stuff' mode right now, supporting multiple hdd locations for large dbs is something I will sneak in, as several people are reaching the limit on this stuff and there is no reason to keep regular files on an SSD.

I will keep client.db in the install dir, but I'll let the user add external directories with GB limits, and set a GB limit on the install dir, and add a daemon to regularly rebalance client_files across those directories.

>>1466

Have a look into FreeFileSync, which does backup maintenance very well. I run a backup every week onto an external usb drive. The inital x-hundred GB run took a long time, but now it takes a few minutes a week. I don't have to worry about hardware crashes, and if there is a house fire or something, I can just grab my usb drive and jump out the window, all important data safe.

>>1463

It isn't supposed to! Usually, the program boots in a few seconds. Initialising the database on first boot usually only takes a second. Do you have a 'crash.log' in your hydrus install directory? Or any errors in install_dir/logs/client.log? Is your hard drive full? Are you running the program from a protected location like C:\program files? Does it work if you run client.exe as an administrator?

>>1482

It looks like they stop serving results after a certain count, like so:

http://www.gelbooru.com/index.php?page=post&s=list&tags=kantai_collection&pid=29988

>Unable to search this deep in temporarily.

I expect they can't/don't want to burn the CPU and memory on caching search results this deep. 20k results is about 475 pages, which I presume is enough for any normal web browsing user.

On OSX, files do not seem to be from the drive when they are deleted from the db, as evidenced by the fact that Hydrus tells me (via a search for system:everything) that it has ~13 GB of files. On the other hand, I am told via Finder that client_files has 36.51 GB worth of files in it.

>>1509

Thank you for this report. If you go services->review services, how much total space does the client think it has for the local files service and the trash service?

And if you go to install_dir/logs/client.log, are there a million errors about deleting files?

Found a small bug that mostly just causes an error, replicable by appending a tag like ":this:" to a file then trying to upload to a tag server.

If this is done locally it only turns the ":test:" tag into an empty namespaced tag like "test:"

Hopefully I do code tags properly:

Exception
The repository encountered an error it could not handle! Here is a dump of what happened, which will also be written to your client.log file. If it persists, please forward it to hydrus.admin@gmail.com:

Traceback (most recent call last):
  File "C:\code\Hydrus\build\server\out00-PYZ.pyz\threading", line 783, in __bootstrap
  File "C:\code\Hydrus\build\server\out00-PYZ.pyz\threading", line 810, in __bootstrap_inner
  File "C:\code\Hydrus\build\server\out00-PYZ.pyz\threading", line 763, in run
    
--- <exception caught here> ---
  File "C:\code\Hydrus\build\server\out00-PYZ.pyz\twisted.python.threadpool", line 191, in _worker
  File "C:\code\Hydrus\build\server\out00-PYZ.pyz\twisted.python.context", line 118, in callWithContext
  File "C:\code\Hydrus\build\server\out00-PYZ.pyz\twisted.python.context", line 81, in callWithContext
  File "C:\code\Hydrus\build\server\out00-PYZ.pyz\include.ServerServerResources", line 510, in _threadDoPOSTJob
  File "C:\code\Hydrus\build\server\out00-PYZ.pyz\include.HydrusController", line 254, in WriteSynchronous
  File "C:\code\Hydrus\build\server\out00-PYZ.pyz\include.HydrusController", line 77, in _Write
  File "C:\code\Hydrus\build\server\out00-PYZ.pyz\include.HydrusDB", line 341, in Write
  File "C:\code\Hydrus\build\server\out00-PYZ.pyz\include.HydrusData", line 1670, in GetResult
    
include.HydrusExceptions.SizeException: Traceback (most recent call last):
  File "C:\code\Hydrus\build\server\out00-PYZ.pyz\include.HydrusDB", line 200, in _ProcessJob
  File "C:\code\Hydrus\build\server\out00-PYZ.pyz\include.ServerDB", line 2559, in _Write
  File "C:\code\Hydrus\build\server\out00-PYZ.pyz\include.ServerDB", line 2009, in _ProcessUpdate
  File "C:\code\Hydrus\build\server\out00-PYZ.pyz\include.ServerDB", line 1490, in _GetTagId
  File "C:\code\Hydrus\build\server\out00-PYZ.pyz\include.HydrusTags", line 106, in CheckTagNotEmpty

SizeException: Received a zero-length tag!


Traceback (most recent call last):
  File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.HydrusThreading", line 233, in run
  File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.ClientGUI", line 1934, in _THREADUploadPending
  File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.ClientData", line 1922, in Request
  File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.ClientController", line 310, in DoHTTP
  File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.ClientNetworking", line 230, in Request
  File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.ClientNetworking", line 179, in _DoRequest
  File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.ClientNetworking", line 615, in Request

>>1528

>>1529

Thank you for this report and the exception.

I recently had the same problem trying to turn the incorrectly-parsed '<' into ':<' with a sibling. I tried to be clever by using '::<', but somewhere in the client gui->client db->network->server db->petition approval chain, the leading ':'s got stripped off again, and I ended up with the invalid '< to <' sibling pair.

I think I know how to fix it. I will work on it this week.

I'll make sure to remove your broken 'test:'-type tags as well.

Not sure if it's a bug or if I've fucked something up but anyway.

Importing tags via gallery download page works fine, but when setting a subscription to download tags it doesn't for some reason.

>>1533

I just tried one, and it worked for me, so I think the code is ok.

Is that import options - tags panel in your screenshot from file->options->default tag import options or services->manage subs? If that is from file->options, please check your subscriptions' specific tag options, for if they were created before you altered your defaults, they will still probably have nothing set. If this is so, you will have to update every existing subscription's tag options manually.

>>1537

It's from services->manage subs.

I didn't do anything but it fixed itself somehow. No idea what caused it, probably just my shit internet though.

>>1549

Resetting the cache worked.

I guess it doesn't check already downloaded files or something.

>>1548

>>1549

>>1550

Thank you for the update. Yeah, unless you tell them specifically to recheck old urls, subs only get files and tags for new urls. The 'get tags even if in db' only means for new urls to the sub that the client has already seen before, either in another sub or a downloader page. I will try to make the dialog and help a bit clearer to better explain how they work.

So,i had the brilliant idea to move 40GBs or 70k Plus images to "system:archive" from "system:inbox". I clicked on the tag search section, expecting to see system:inbox 0 and system:archive 70k, only to see that neither sys:inbox or sys:archive predicates were there. At the very least "system:everything" shows an accurate count. Is this an ill omen for my beloved database?

>>1631

(me again) Forgot to note that there were no errors reported by the client. Also I figured a screen of the problem would be better.

>>1631

>>1632

>>1642

>>1643

It has been 20-something months since I had inbox zero, so I forgot this happened! Yeah, if num_archive or num_inbox have the same value as num_everything, the program won't include system:archive or system:inbox in the system predicate list. Perhaps this isn't ultimately helpful. Maybe I could have an indicator for inbox zero status, or perhaps I could make the predicate hiding optional?

>>1649

I'm not too certain. It made sense to me after I found out. I suppose optional, if better over all. After I ran into the "missing" system predicates, I did go to settings to see if I had butchered anything in there. Then I posted of my shame of finding out later.

Trying to add pixiv tags and i constantly get this error when downloading initially, have to re-do several times to full get al lthe images from some tags.

Exception
The subscription Pixiv Little Busters! encountered several errors when downloading files, so it abandoned its sync.
Traceback (most recent call last):
  File "C:\code\Hydrus\include\ClientImporting.py", line 2306, in Sync
  File "C:\code\Hydrus\include\ClientImporting.py", line 2088, in _WorkOnFiles
Exception: The subscription Pixiv Little Busters! encountered several errors when downloading files, so it abandoned its sync.

>>1656

Thank you for this report. I think my new subscription error protection code is being oversensitive for pixiv ugoira, which it wasn't detecting and safely ignoring correctly. I have fixed this for v188.

Just to check that you have the same problem, if you go services->manage subscriptions->your pixiv sub->import status icon button (just above the 'reset cache on dialog ok' button) and then right click on a failed download and hit 'copy notes', do you get something like this?:

Traceback (most recent call last):
  File "C:\code\Hydrus\include\ClientImporting.py", line 204, in _WorkOnFiles
    tags = gallery.GetFileAndTags( temp_path, url, report_hooks = [ self._file_download_hook ] )
  File "C:\code\Hydrus\include\ClientDownloading.py", line 1400, in GetFileAndTags
    ( image_url, tags ) = self._GetFileURLAndTags( url, report_hooks = report_hooks )
  File "C:\code\Hydrus\include\ClientDownloading.py", line 1388, in _GetFileURLAndTags
    return self._ParseImagePage( html, page_url )
  File "C:\code\Hydrus\include\ClientDownloading.py", line 1362, in _ParseImagePage
    image_url = original_image[ 'data-src' ] # http://i3.pixiv.net/img-original/img/2014/01/25/19/21/56/41171994_p0.jpg
TypeError: 'NoneType' object has no attribute '__getitem__'

I don't have tracebacks for these crashes as they don't produce them but Hydrus hard crashes when changing themes and when shutting down on Windows. Not a huge issue, but quite indicative that something nasty is happening.

The shutdown crash is an access violation and probably has to do with Hydrus having a long shutdown? It's hard to capture as Windows chugs on with shutdown regardless.

Whereas the crash on theme change produces this:

Problem signature:
  Problem Event Name:	APPCRASH
  Application Name:	client.exe
  Application Version:	0.0.0.0
  Application Timestamp:	00000000
  Fault Module Name:	wx._core_.pyd
  Fault Module Version:	0.0.0.0
  Fault Module Timestamp:	5477a7bd
  Exception Code:	c000041d
  Exception Offset:	0000000000007216
  OS Version:	6.1.7601.2.1.0.256.48
  Locale ID:	1033
  Additional Information 1:	ec95
  Additional Information 2:	ec9599b53bd8e876d8f996dbb20361c0
  Additional Information 3:	82e6
  Additional Information 4:	82e668c667968b2445df6a18e0b11c4e

>>1661

That's odd. Do you get errors like that doing anything else in the program?

That error usually means that the connection broke because of someone else's fault, be that pixiv closing it rudely or your firewall kicking in, or something else the client can't determine. It is usually rare, like if a server is overloaded or your router has a hiccup. Normally, if a server doesn't want to talk to you, it gives you a nice html page saying "Too much bandwidth" or whatever with a 401 error code or whatever, but this is just an abrupt end of conversation.

Are you running on slightly unreliable wifi or similarly unusual network setup/internet connection? How many files have you roughly downloaded from pixiv–a few thousand, or much more? Could some bandwidth firewall on their end be cutting you off?

I just downloaded a couple hundred files and didn't get this once, so I don't think some bad cod of mine is causing it.

In any case, I will make a note to handle this error better for when it does happen. I'll wait a sec and try again up to two times or something. Thank you for reporting it.

>>1664

I am sorry you are having this problem. If you don't mind answering:

Which version of Windows are you using? Are you running any unusual window themes or managers?

Do you have a particularly old computer? Does hydrus often lock up for a few seconds while you are using it normally?

Do you have hydrus set to run large processing jobs when it shuts down (under file->options->maintenance and processing)? If you do, does the client shut down ok if you turn them off?

Have you ever had a crash or odd flickery gui behaviour when starting the client?

If you go help->debug->db profile mode before you try to shut down, do you get any tracebacks printed to your log (it is install_dir/logs/client.log)?

When I put the release together tomorrow, I will make a special debug version for you that might spell out more what is going on here. I will post the link in this thread and some instructions on how to run it–if you have some time to keep working on this, please check it out!

>>1665

>Which version of Windows are you using? Are you running any unusual window themes or managers?

I'm currently running Windows 7 SP1 x64, this happened on Windows 10 x64 as well.

I am using a modded theme but the same behavior happens with an unpatched uxtheme.dll and switching between stock themes.

I am currently using Dual Monitor Taskbar on Windows 7 but I was not using this on Windows 10. Other than that no unusual window managers.

>Do you have a particularly old computer? Does hydrus often lock up for a few seconds while you are using it normally?

No, specs are in pic. Hydrus does not lock up unless I am uploading tons of petitions and files at the same time.

>Do you have hydrus set to run large processing jobs when it shuts down (under file->options->maintenance and processing)? If you do, does the client shut down ok if you turn them off?

To clarify when I said shutdown I meant Windows power off, the client closes properly if I manually close it.

I have hydrus set to "run jobs on shutdown if needed, but ask first".

I managed to capture a screenshot of the access violation, attached.

>Have you ever had a crash or odd flickery gui behaviour when starting the client?

Nope, none so far.

I turned on db profile mode and performed each crash, hydrus seems to write nothing to it's log when crashed in either way (other than the usual startup stuff).

I should have time to mess with the debug version when posted.

>>1667

Ah, thanks for the update. I misunderstood your problem. We'll skip the debug release for now.

I have never tried to exit the client via windows shutdown before! I just tried it on my Windows 7 dev machine, and it was ok, but my OS X laptop had a similar error, so it looks like I can probably reproduce this problem myself. I expect I am not handling some special 'shut down right the hell now' OS event and opening the splash window when no new windows are allowed or something. I will play with it this week and see if I can fix it my end.

giphy never works

>Gallery 404

I've tried

#serial_experiments_lain

#serial experiments lain

serial_experiments_lain

serial experiments lain

and

anime

#anime

I'm using the linux version, btw

So I tried to install hydrus for the first time today (v188, of course.) and the installer didn't seem to want to come up for me. Task manager says that the process is running, but there's no window for the installation in sight. I gave ending the process and then running it again as an administrator a shot, but all that did was freeze the folder that I had the installer application in. Weird stuff.

I ran an installer for another program and that worked just fine. I downloaded the past couple versions of hydrus and tried to run them, didn't work either. Honestly, I'd just love to know whether other people have been having this problem or if it's on my toaster's end.

>>1683

Thank you for the report. I added giphy support a long time ago, so I expect their API has changed or disappeared. I will look at it, and if there is an easy fix, I will switch the url over or whatever.

>>1686

That sort of thing is usually from overzealous anti-virus throwing a false positive. If you do run an anti-virus like Avast, see if it has a log you can look at, as there might be a note about some file it blocked. I think install_dir/bin/upnpc_win32.exe was causing that problem for some people a little while ago.

Also, please try downloading the .zip 'extract only' release and see if you can extract it to somewhere simple like your desktop.

https://github.com/hydrusnetwork/hydrus/releases/download/v188/Hydrus.Network.188.-.Windows.-.Extract.only.zip

If that stops extracting on a certain file, it is probably that causing the problem, and if it works, then we know it is the installer itself. If it does extract ok, then try double-clicking client.exe, which should be in the base directory.

Let me know how you get on!

>>1700

Maybe–are you using your premium user/pass inside hydrus? I don't know much about Pixiv, but perhaps they have some browser filtering for premium accounts or something? Do you get access to more stuff as a premium person, so it might be specifically to stop scraping their premium stuff? Still, I would expect they would return a nicely formatted "403 Forbidden" or something if that were the case. I seem to remember a nice error from them when I was trying to download fullsize images without the correct http referral request headers from the image page.

At the moment, hydrus doesn't pretend to by Firefox or anything. It just says "I'm hydrus/188, gimme stuff". Maybe a 'pretend to be a browser' checkbox is something to add in future.

I have updated the socket-level error catching for tomorrow's release, by the way, so the problem might disappear. The client will wait a bit and try the request twice more before escalating the exception.

If it still doesn't work, you could try generating a non-premium throwaway pixiv account just for hydrus to use.

>>1683

I had a look, and unfortunately the simple open API that giphy used to provide has been replaced with a closed one. I don't have time to write a new manual parser, so it will have to wait until I extend the downloader engine enough that people can create their own.

I have removed giphy from the list of download pages for tomorrow's release.

It seems the last update broke the counting of files in a given collection in the Linux build for me, every "collect by [namespace]"action results in thumbnails displaying a zero for their file count (or nothing, if it's just one image/file)

>>1728

Thank you for this report! It is fixed for next week.

>>1765

I am very sorry you are getting this. I have been talking with someone in the release thread about a very similar problem, starting here: >>1725 .

Have you used previous versions of hydrus on Win 10, and did they work ok?

Are you running video or game software in addition to hydrus when you get these BSODs?

Is the client booting or shutting down when the BSOD occurs, or can it happen just when you are browsing? Has it ever occurred when hydrus is idle?

If you navigate to C:\Windows\SysWOW64, is there a sqlite.dll file? How big is it?

>>1777

>>1768

Rolling back to 188 returned this error when I chose system:everything

>>1772

Is that error happening on files you just downloaded, or on files you have had in the db a while? That error means hydrus looked for the file but could not find it.

What was your recent reformat? Did you move a lot of folders around?

Have you set external file storage locations in options->file storage locations? Could those locations have recently changed, or has hydrus's install location recently changed? If you look at the locations listed in the options dialog, are they what you expect, or is a drive letter or path incorrect?

If your external locations are wrong, fix them in the dialog and then go database->maintenance->rebalance file storage. The client should repair its understanding of where everything is.

>>1778

Unfortunately, v190 changed the way the db tracks files, so you can't downgrade v190->v188. You will just get those file table errors whenever the client tries to cross-reference files with anything. If you have a backup of v189, you can downgrade that.

Other than the popup errors, is that v188 code stable at least? If so, that proves it is v189 causing the problem.

>>1776

>>1777

Thank you for these updates. The missing sqlite.dll is actually good news–apparently BSODs can occur when different versions of sqlite.dll conflict in Win10, and SysWOW64 is one potential location for that.

It seems something I changed in v189 is causing the problem. I suspect it is one of:

The new analyze stuff I wrote.

A new pauser object that makes daemons wait while the gui catches up.

Setting the db to TRUNCATE journalling mode before a vacuum call.

Changing the db page size for Windows dbs during a vacuum call.

Increasing the db cache from 10MB to 50MB.

Luckily, most of this stuff only happens in the daemons (the workers that do maintenance jobs in the background). They usually kick in a few seconds after a launch and just before an exit as well.

So, here is a special build:

http://www.mediafire.com/download/dxkvm4ny29z3ab7/Hydrus_Network_v190_with_no-daemons_-_Windows_-_Extract_only.zip

That is essentially v190a. If you start it from the command line like this–

client -no-daemons

–it will not run any daemon jobs. The db won't maintain itself, and repositories won't synchronise themselves. If you have the client set to run maintenance on shutdown it might still cause a problem, however, so please switch that off. Please let me know if that is stable for you. If it is, you can keep using it for now and we can start narrowing down exactly which call is causing Win 10 trouble.

If you still get BSODs, then please let me know as usual. You might also want to try running the client in Windows Vista compatibility mode, as that temporarily fixed a possibly similar shutdown-crash bug recently.

Just got an error while trying to vacuum the database saying that the database or hard drive was full. However, the drive that the program is on (H), has 70gb free. When I copied the traceback info, this is what it gave:

 OperationalError
database or disk is full
Traceback (most recent call last):
  File "C:\code\Hydrus\include\ClientDB.py", line 6823, in _Vacuum
OperationalError: database or disk is full 

Checking database integrity did not return any errors.

>>1786

I remember an issue some (I think Linux) users had a while ago where they had enough space on their hydrus partition, but the temporary vacuum file (which ends up about the same size as client.db, which for most people is ~3GB right now) was going through or being stored in or was in some other way related to their OS temporary folder, which for them was capped to 500MB or something because they were doing Linux ramdisk space magic.

Since your C drive has <3GB of space, maybe something similar is happening? If you temporarily create a few gigs of space on C and try a vacuum, does it work then?

If not, you can try to do a vacuum from console, which should give more error information. Close the client, go to install_dir/db and double-click sqlite3.exe.

A terminal window will open. Type exactly this:

.open client.db

.log stdout

VACUUM;

.exit

Note that VACUUM has a semi-colon after it.

While it works on that, you can watch the db journalling file grow inside your db dir. It will take several minutes, but the client.db-wal or client.db-journal file (whichever you have) should grow to ~98% of client.db's size, and then when it is done, the files are supposed to 'swap' and the journal file is deleted. You will likely get the same error again, though.

It will hopefully say something like '(522) blah blah disk full'. The number inside the parentheses is a useful error code.

Extra questions, if you don't mind:

How many files do you have in your hydrus client, roughly?

How big is your client.db file?

Do you know how long your client spent running its vacuum? Was it likely a few minutes, or could it have been much longer?

>>1788

>>1790

That sounds promising! If you have the time and would like to continue testing, please try going database->maintenance->vacuum, and see if it causes BSOD.

>>1797

Thank you for the update. I don't really know what is going on here, but at least we know how to fix it. I will update my db help files.

>>1799

When you backed up, did you make a full copy of client.db and the client_files directory? That error suggests you have lost some files somewhere. Perhaps your backed-up client_files directory was an older version than the client.db, or maybe some transfer step didn't complete fully?

Inside client_files should be 256 folders, 00 to ff. Do the folders near the top have roughly the same number of files as the ones at the bottom? If you have a program like SpaceMonger or WinDirStat to view your install directory as a treemap, do all those 256 folders seem to have roughly the same size?

I am not at my dev machine now, so I can't remember if I added this in v190 or if this is pending in v191, but I have improved the error handling for this situation so it will now say the exact path that was missing. When you start seeing those better errors, see if the missing files all start with 'd4', like, 'install_dir/db/client_files/d4/d4123456…' or similar. That might help figure out what happened.

>>1800

If you run that version with -no-daemons, do you get the BSOD? You can do this by running from command line. Shift-right-click your install_dir's explorer window and hit 'open command window here'. Then type:

client -no-daemons

You can also set up a shortcut to client.exe to do the same. That will launch the client without some background processes, like the one that syncs subscriptions. I think it is exactly those background processes that are causing the BSOD. If you can run that special build without daemons and without BSOD, that confirms it. Once we know that, we can start narrowing down exactly which one is causing the problem.

Have you ever had a thumbnail error before? Was the exception traceback written to the bottom of your install_dir/logs/client.log?

Uh, I left hydrus running for a few hours and this was here when I came back.

Should I be worried? Here's the traceback (I think each one is the same) http://pastebin.com/ehw6QDvH

I'm not sure if this counts as a bug or should be posted under feature requests, but right now when scraping a given pixiv artist by id the client seems to skip any entries that are not single pictures.

I wouldn't expect hydrus to handle ugoira, as those are a real pain, but it also skips any and all grouped pictures/picture galleries because it finds the mime to be 'uninteresting'

This makes scraping somewhat problematic, as many artists over there love uploading in bulk, and going through the 'file import status' entries one by one to grab missing files takes a lot more time than just doing the entire thing by hand and importing afterwards

Am I maybe not using the downloader correctly?

>>1815

>>1824

>>1825

>>1826

>>1827

>>1828

>>1830

Thank you, this is very helpful. It sounds like the database maintenance is probably ok, but something in file searching or general result page maintenance is certainly triggering it. This is odd, because I didn't make much change to that code in v189, and it is unusual that something so high-level could cause cause BSOD. I rewrote some behind-the-scenes media management code to be more efficient when new thumbnails were added to an existing result set, like in an import window, but this would not be called by a normal search page.

There is an outside possibility that it wasn't something I added in v189, but instead Microsoft pushed an update at around that time that began having a problem with something I previously added. A web search for 'win 10 update bsod' gives copious examples of botched security updates causing problems.

So, if you still have time and enthusiasm, you could try running a v188 client for a short while to check that that definitely works ok. Because of the recent file changes, you can't downgrade from v190, so please create a new v188, import more than 256 files, and then run several system:everything queries in multiple pages. The link is:

https://github.com/hydrusnetwork/hydrus/releases/download/v188/Hydrus.Network.188.-.Windows.-.Extract.only.zip

As it happens, I am in the midst of a big computer/network switcharound right now, including moving my hydrus development to a new Win 10 computer next week. I am hoping I will get this BSOD on my new machine so I can quickly figure out what is doing this and fix it. I will be reinstalling several other computers and playing musical chairs with about eight hard drives next week, so am not planning to do much hydrus work or put out a normal release, but if I can repeat this BSOD, I will put out a release just for that on the 27th.

>>1815

That thumbnail error suggests you may have other problems as well. Not only was the thumbnail for that file malformed, but the client failed to delete it from your hard drive. If you go to D:\hydrus network\db\client_thumbnails\31, does the 319d3c… file exist? There should be two–one with the whole hexedecimal hash as its filename, and another with '_resized' at the end. Maybe the resized one has size 0KB.

If you have them and they seem ok, copy them to your desktop and rename them to put .jpg on the end. Then try double-clicking on them to open them in your default image viewer. Do they load?

If you go to …\db\client_files\31\319d3c…, does the full file exist there? What is its file type, and if you double-click it, does it load ok in whatever your default program is?

I've improved that error to say more in v191. When you are ready to try normal browsing again without risking a BSOD, you should get a second traceback with the exact error the OS is giving when trying to delete that file.

You can quickly look up that file by searching for system:hash=319d3cb16263ac431a0e87e187158b2ac9191b9953213410db2ae31524627d9f .

>>1827

Would you estimate that the BSOD occurred at the beginning of the search, or somewhere in the middle (where it says 'Loading… 256 of 11,500', or the final step, where the thumbnails pop in on the search page?

>>1829

This could be serious. It might be that your database is broken, or it could be a false positive error from something else. If the client is running, shut it down now. Don't open it again until we are sure this isn't a big problem.

Have you had any recent power cuts or other problems with your hard drive? Is it particularly old or fragged or full up?

Go to install_dir/db and check the file 'help my db is broke.txt', where I've written a simple guide on checking what is going on.

Let me know how your chkdsk goes. You might also want to launch sqlite3.exe and run:

.open client.db

.log stdout

SELECT * FROM files_info LIMIT 100;

ANALYZE files_info;

ANALYZE;

.quit

Some of those queries may give more detailed errors that will help figure out what is wrong.

If this isn't a simple problem, I can help you try to recover what's inside your db.

>>1831

At the moment, I'm afraid pixiv manga (and any other single page -> multiple file action) isn't supported. A future version of the downloader engine will support it. When the next vote on what I should work on comes around, please vote for improvements to the gallery downloader.

>>1834

>Have you had any recent power cuts or other problems with your hard drive? Is it particularly old or fragged or full up?

The hard drive is brand new and still mostly empty. No power cuts that I know of. chkdsk reported no errors.

Running those queries showed this.

The client isn't running right now, but I was using it for several hours after I got the error with no more issues, and this is actually the second time it's appeared. I ignored it the first time because I'm dumb.

>>1838

Alright, I followed the instructions in the text file. Got these errors when I did an integrity check, and a few errors when attempting to clone the database.

I'm not sure how serious these are. Luckily, I did make a backup a week or two ago, although I'm not sure when these errors were first made.

>>1837

I am not totally sure what's going on there, then. I did find a typo in a different bit of that code, so maybe there is another one. Let me know what error you get in v191 and beyond.

>>1838

>>1840

You may be in luck. Those integrity errors are all in an index, which is internally duplicate data.

I also just ran a .clone and got the same errors, but the file was created ok, and the cloned tags_fts4 seems to have the correct content. I guess .clone has a minor hiccup on virtual fts4 tables. I don't know why it is trying to copy sqlite_statx either–it is correct to not ultimately copy that info over.

I suggest you hang on to your backup db and the old broken client.db and try using your new clone (assuming it got created despite those errors). If the client opens ok, go help->debug->force idle, which should trigger the maintenance routines that were throwing the error last time. If they proceed without another 'malformed' error, I think you are good. If not, then we'll look at it closer.

If you prefer, you can also test it manually by running the ANALYZE; call again on your new clone through sqlite3.exe.

If you only lost a sector or page related to an index, then nothing is lost. It is worrying that those rows are gone, though. Something went wrong with your db, possibly a bug in sqlite or more likely a problem with your hard drive that randomly zeroed 4096 bytes or similar. Since it is new, I am not sure why that would happen. Make sure all the cables are well seated, I suppose, and hang on to your backup for a little longer. Keep me updated if anything new happens!

I tried updating hydrus using the windows installer but now the client wont even start! I just get this error

I was about to format the PC, so I moved the hydrus folder (the one with .db, files and stuff) to my external HDD, I reinstalled my OS, hydrus and restored the database using that folder, it went smoothly, searches work and everything, but whenever I try to open a file pic related happens.

Basically the file doesn't open, I can only perform searches and see the thumbnails, if I touch any file, I'm get flooded with shitloads of error messages.

Also, if I actually open the file, 2nd pic related happens (it shows nothing other than the name and tags)

Here's the traceback

NotFoundException
File not found in path + /home/wsy/Pictures/hydrus/df/dfef7c0603a13904e9dbec0903c87909b3e228c2aebcee20840da19f586ea688.jpg!
  File "/usr/lib/python2.7/site-packages/wx-3.0-gtk2/wx/_core.py", line 16762, in <lambda>
    lambda event: event.callable(*event.args, **event.kw) )
  File "/opt/hydrus/include/HydrusController.py", line 210, in ProcessPubSub
    try: self._pubsub.Process()
  File "/opt/hydrus/include/HydrusPubSub.py", line 127, in Process
    callable( *args, **kwargs )
  File "/opt/hydrus/include/ClientGUICanvas.py", line 1616, in FocusChanged
    if page_key == self._page_key: self.SetMedia( media )
  File "/opt/hydrus/include/ClientGUICanvas.py", line 1220, in SetMedia
    self._media_container = MediaContainer( self, self._image_cache, self._current_display_media, initial_size, initial_position )
  File "/opt/hydrus/include/ClientGUICanvas.py", line 3150, in __init__
    self._MakeMediaWindow()
  File "/opt/hydrus/include/ClientGUICanvas.py", line 3217, in _MakeMediaWindow
    self._media_window = self._media_window = StaticImage( self, self._media, self._image_cache, media_initial_size, media_initial_position )
  File "/opt/hydrus/include/ClientGUICanvas.py", line 3477, in __init__
    self.EventResize( None )
  File "/opt/hydrus/include/ClientGUICanvas.py", line 3563, in EventResize
    self._image_container = self._image_cache.GetImage( self._media, ( my_width, my_height ) )
  File "/opt/hydrus/include/ClientCaches.py", line 911, in GetImage
    image_container = ClientRendering.RasterContainerImage( media, target_resolution )
  File "/opt/hydrus/include/ClientRendering.py", line 94, in __init__
    RasterContainer.__init__( self, media, target_resolution )
  File "/opt/hydrus/include/ClientRendering.py", line 76, in __init__
    self._path = client_files_manager.GetFilePath( hash, mime )
  File "/opt/hydrus/include/ClientCaches.py", line 340, in GetFilePath
    return ClientFiles.GetFilePath( location, hash, mime )
  File "/opt/hydrus/include/ClientFiles.py", line 194, in GetFilePath
    raise HydrusExceptions.NotFoundException( 'File not found in path + ' + path + '!' )

What can I do?

I have about 8k files and I would like to keep them as organized as before I formatted the PC.

I have Never seen any error messages, yet when I start a remote tag account with the base key, nothing happens. I can check the credentials and it says everything looks okay, but nothing ever downloads.

How do I force the program to start downloading the tag data for the meta data? I have never gotten past this stage and it makes the program basically .

>>1860

I think you have hit the 32bit->64bit jump problem. You will need to perform a clean install. Please check the v186 release post here:

http://8ch.net/hydrus/res/1589.html

>>1873

You could go database->maintenance->check file integrity, but I just looked at it, and it doesn't print any information about missing files–it just removes them from the current db file table. I'll update it for v192 to print missing paths to the log.

Until then, please make sure your hard disk is ok, and maybe move hydrus off that partition if it is likely to do that in future. Randomly deleted files does not sound good! Check 'help my db is broke.txt' in the install_dir/db directory if you haven't already. Also, if you have a backup, you could try merge-copying your old client_files directory into your new one, which would recover most/all of the files.

>>1876

Thank you for this report. It is fixed for v192. Please let me know if you still get it, or anything else!

>>1911

>>1912

I am sorry about this. I've been thinking about this problem, but haven't come up with a clever way to heal these sorts of problems automatically.

I will add an explicit check at the beginning of every client boot, where it will say "Hey, I think your external storage location x is moved/missing!'.

>>1922

It is only supposed to happen when your client is 'idle', which you can customise in file->options->maintenance and processing. You can force it via the help->debug menu, but this is not recommended, especially for new users. For most people, the initial sync takes a week or two, so if you leave the client alone for six hours and come back, you should see some progress.

If you have some big defrag program running, though, or you use the same computer to browse or play games, then the client might not do anything because it will assume the system is busy and will not want to kill your cpu. Again, check the options dialog and set whatever idle options work for you. If you would really prefer to have manual control, you can also set the client to only do big maintenance jobs on shutdown.

>>173

My client recently has stopped responding at a relatively high rate, when running Hydrus as an administrator this doesn't happen, but I get pic related as errors

>>1953

Thank you for this report. This bug is fixed in tomorrow's release. Please let me know if you still get anything like it in v192.

>>1811

Sorry for the late reply lost internet for 2 weeks, It seems I've only got 87 folders in therer, looking at my backup folder theres only 171 in there, safe to assume i've screwed up the back up?

If that's the case should i just nuke everything and start over i guess?

>>1971

Unfortunately, I think so. Did you ever add an external storage location under file->options->file storage locations? It is worth checking that options panel anyway, just in case. It could be those folders got sent somewhere else. I think that's the only way the client would ever move/delete a folder from client_files.

If it looks like the extra folders were lost in your backup/restore process, then I think your best bet is to gather all the surviving folders you have, start a new client, and then try reimporting them. You should recover about two thirds of your stuff.

I use FreeFileSync for my backups, and I find it works very well.

>>1993

>>2011

Hey, sorry for the late reply here. I missed these the first time around because of all my recent IRL stuff.

>>1993

I am not sure what this is. It might be a code logic problem that is literally locking up the program, but I expect it is hard drive lag that I am not dealing with well.

The hydrus db is too big right now, so sometimes when transactions commit or the db engine decides to clear out its journal, it can take a very long time, especially on systems with very fragged drives or otherwise unusual storage. You may not see hard drive activity, but the db could be waiting on Windows to clear out some temporary file or give up some exclusive lock. The gui works until it needs the db, at which point it waits infinitely in these cases, locking up until the db can process new jobs, which is frustrating. I plan to split up the db file over the coming weeks to reduce this lag.

If you haven't checked out my lag help page, please do:

http://hydrusnetwork.github.io/hydrus/help/reducing_lag.html

Do you have an aggressive defragging program or anti-virus, or something else that might be choking the client.db file every time it grows a little?

Try doing an import, and when it locks up, open install_dir/db. Do you see a client.db-wal or client.db-journal file growing very slowly? Usually about 1MB per minute. If it is doing stuff to that, then the client is clearing out some stuff, even if its CPU looks like 0.3%.

Give the client about twenty minutes, and it may have unfrozen itself. See if it freezes up again, or if this is a one-time thing. If it keeps freezing, then follow the 'db profile' instructions on that lag help page and send me whatever profile seems to span the frozen step, and I'll be able to pinpoint more what is going on.

If it is still stuck after twenty minutes, then let me know and we can look at it closer.

>>2011

It looks my UPnP parser doesn't understand what your router is returning.

Please go to your install_dir and shift+right-click on the 'bin' directory and select 'open command window here'. Then type in:

upnpc_win32 -l

That's a lower case L. This is the query hydrus is making to ask your router a whole bunch of info about your network. Somewhere in there should normally be a line that says:

ExternalIPAddress = [your ip]

Do you have that? If not, what do you get?

If you do get it, and there seems to be a reasonable number afterwards, is the next line ' i protocol exPort->inAddr:inPort description remoteHost leaseTime'?

You can send me the whole output if you like, but don't post it raw here, because it will have lots of your private information. You can email it to me, or carefully censor all the 192.168.0.1 stuff in photoshop before you post it here.

You can send the console output to a text file by going:

upnpc_win32 -l > upnpc.txt

>>2050

Since hydrus can't figure out your external IP, you will have to provide it manually. I have added an option for that in file->options->connection. Put in your real external IP, or host if you have one, and the client will use that when it generates external urls rather than trying to figure it out for itself. Let me know if it doesn't work.

If you don't know your IP, just throw it into a search engine:

https://duckduckgo.com/?q=my+ip

They don't typically change all that often, but if yours does, look into services like no-ip.com.

I'll make a note to improve that original error message.

>>2087

>>2089

Yeah, this is usually due to siblings, which are not collapsed in the db, where autocomplete counts are cached, but are in the gui, where the true results are obviously shown and tag counts are generated live. I expect there are a bunch of 'character:hatsune miku (character)' tags in your collection that are normally being hidden.

I want to address this in the upcoming service cache layer I'll be adding. I see two possibilities:

1. Collapse the sibling counts in the cache layer (probably complicated to code correctly, but speeds some things up and will give 100% accurate a/c counts).

2. Propagate any uncertainty up to the gui, so when a tag count is imprecisely aggregated, I can say "character:hatsune miku (≤168)".

I suspect, since I want to break this big job up anyway, that I'll start with 2 and slowly migrate to 1.

>>2088

I think your client might be freezing because of a vacuum. Both vacuum and repo sync checks are triggered by the 'changing to idle' event, and sometimes they compete. If the vacuum is stomping all over your repo sync, it would appear to freeze, when really it is waiting for the db to free up.

Try going services->pause->repo sync and then help->debug->force idle. If your client freezes again, it is probably vacuum doing it. This maintenance operation is now very laggy, often taking ten minutes to complete. If your client doesn't unfreeze after say twenty five minutes, you can kill it in task manager, then restart it and go ''file->options->maintenance and processing' and stop auto-vacuum from occuring.

Vacuum should be a much smaller problem once I have pulled the big client.db file apart as I plan.

If it isn't vacuum causing it, let me know and we can look closer at this.

>>2096

This is only odd because 1204 is just a tiny 58-byte bit of near-empty json that your db should quickly process like the thousand-odd before it. My clients are up to 1207 right now, so it seems to be something particular to your setup. I cant help thinking it is still something else that is lagging, stopping the repo from syncing, but I can't guess what. Do you have any very large import or export folders, or subscriptions or something?

Anyway, please check your log, at install_dir/logs/client.log. At the bottom of that file, see if there are any error logs that might be related. You should see the successful vacuum right at the bottom, so just a bit up. A successful repo process is reported like this:

2016/02/20 17:05:02: repository synchronisation - read-only art file repository - finished

1 updates downloaded, 1 updates processed, and 0 files added

I presume you will see the end of those around that date.

Of course, because you are getting a freeze, it may be that any error isn't ultimately being caught, or the client doesn't have enough time to report it.

If you don't find anything good in the log, try going help->debug->db profile mode before forcing idle mode again to trigger repo sync. This will try to spam more detailed information to the log.

Also, if you open task manager (Ctrl+Shift+Esc if you are on Windows) when the client is frozen, does the client have any CPU or HDD activity or is it 0% for everything?

>>2104

Let me know if it acts up again, especially after I break the db up.

>>2117

Thank you for this report. I think I have fixed it for next week, but I am not sure, so let me know how you get on in v195.

>>2161

I've had something like this happen to me when I had review services open, and then the perform a service-wide operation dialog on top of that. If I started a big operation that spawned a popup message, windows would bring the main gui to the top to show the popup, obscuring the review services frame and its child dialog, but since the dialog was still open (modal), I could not click on the gui at all. I was typically confused by this because I would be coming back to the client after waiting twenty minutes for the big job to complete, and then didn't realise the dialog was still open in the background. When this happens, I have to alt-tab to bring the dialog back to the front.

Could something like that be happening to you? Could manage tags somehow still be open as a child window of the media viewer, behind the main gui? Or maybe off-screen?

When it next happens to you, please try alt-tabbing to see if there is an unexpected dialog open or if you see another tab in your taskbar to suggest similar. Let me know if you can reliably induce it, and I'll see if I can fix it.

On hydrus v195 I can't import this file, probably because the filename contains UTF-8.

Traceback:

UnicodeEncodeError
'ascii' codec can't encode character u'\u2206' in position 38: ordinal not in range(128)
Traceback (most recent call last):
  File "/opt/hydrus/include/HydrusThreading.py", line 274, in run
    callable( *args, **kwargs )
  File "/opt/hydrus/include/ClientGUIDialogs.py", line 1682, in THREADParseImportablePaths
    file_paths = ClientFiles.GetAllPaths( raw_paths )
  File "/opt/hydrus/include/ClientFiles.py", line 93, in GetAllPaths
    if os.path.isdir( path ):
  File "/usr/lib/python2.7/genericpath.py", line 49, in isdir
    st = os.stat(s)
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2206' in position 38: ordinal not in range(128)

>>2174

I have had a closer look at this, and I think I have it figured out now. I noticed that the sibling pair momiji->character:momiji (ninja gaiden) exists on my PTR, and both some bad sibling substitution logic and some bad gui update code were also incorrectly filtering and displaying these sorts of results.

I don't like this sort of too-simple sibling pair because it can cause problems like this. I don't usually approve them, so I must have missed it. I have rescinded it, so the general bad logic here should be fixed for next week, and the unhelpful 'momiji' sibling should disappear for you once you are synced up to 1,341 on the PTR.

>>2194

Thank you for this report. My new Win 10 dev computer seems to be giving me paths already in unicode, so I can't reproduce this myself, but I think I see what is happening. I think I have fixed it for next week. Just to check, can you please test that:

You get the error when you drop the file onto the main gui, spawning the import dialog.

You don't get the error when you drop the file onto the import dialog itself.

Which OS are you using?

Something curious happened to me today. I imported a huge folder, then I decided to reimport it in order to handle it better (yeah, poor decisions). So since my images were already correctly named, I decided to use a regex to capture the filename without the extension and they were parsed correctly.

Issue: some tags apper as duplicated even though they are not, and those that appear only once are not displayed when selecting an incorrect file.

Restarting the client recognized the accurate number of files and all the tags. However, repeating the process with another batch of files produced the same inaccuracy. Tags are accurate on all files, the problem is in the display on the "selection tags". If you need logs, just ask away, although I think it's generalised.

>>2204

>>2206

>>2207

Thank you for these thoughts. It is interesting and useful to see how these things work in the real world for people with different workflows.

I had forgotten to add a thumbnail check for immediate sync, for instance, as I normally use it for tags. Immediate sync also only adds the content from last known update end timestamp->now, and does not immediately process the intervening updates, as I always keep my admin client synced through idle processing, rather than shutdown processing.

I will keep this stuff in mind and try to make some easy fixes, but I think you are right in that I will have to overhaul the server to better support your sort of fast-update local network situation. It may be that I should just write an 'ftp server' sort of service that churns through more CPU and bandwidth but is always immediate.

I have also been thinking recently of combining the file and tag repos together, so there is only a 'repo' that can store or spit out any sort of content, and allowed content types will be set through user permissions.

I believe IPFS keeps its own cache of your files, so it also dupes anything you are sharing. I think it is under C:\Users\You\.ipfs on Windows. It actually transfers files in pieces that it wraps in JSON or something, so I think it caches those there.