[ / / / / / / / / / / / / / ] [ dir / agatha2 / animu / arepa / centraca / general / mde / tacos / vg ]

/hydrus/ - Hydrus Network

Bug reports, feature requests, and other discussion for the hydrus network.

Catalog   Archive

September 2018 - 8chan Transparency Report
Name
Email
Subject
Comment *
File
Password (Randomized for file and post deletion; you may also set your own.)
* = required field[▶ Show post options & limits]
Confused? See the FAQ.
Embed
(replaces files and can be used instead)
Options

Allowed file types:jpg, jpeg, gif, png, webm, mp4, swf, pdf
Max filesize is 16 MB.
Max image dimensions are 15000 x 15000.
You may upload 5 per post.


New user? Start here ---> http://hydrusnetwork.github.io/hydrus/

Current to-do list has: 1,504 items

Current big job: last touches of downloader overhaul, then login manager


YouTube embed. Click thumbnail to play.

2b090d  No.10273[Reply]

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v326/Hydrus.Network.326.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v326/Hydrus.Network.326.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v326/Hydrus.Network.326.-.OS.X.-.App.dmg

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v326/Hydrus.Network.326.-.OS.X.-.Extract.only.tar.gz

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v326/Hydrus.Network.326.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v326.tar.gz

I had a great couple of weeks. I was able to get some good prototype login stuff out for advanced users to play with and a bunch of other stuff besides.

Some login behind-the-scenes changed this week. Please report any problems you have with hanging/failing Pixiv, Hentai Foundry, or Hydrus service (i.e. the PTR) jobs.

login

AdvanPost too long. Click here to view the full text.

131ba2  No.10274

any benefit to going to 3?

or is it just because 3 is newer?

On a side note, one of my sub I have check every 6 hours keeps having the 6 hours changed to 6 minutes. ill probably have a bit to say later on when I get more time with it, but this is an immediate thing I notice. it has happened before, but I assumed it was my own error that time.




File: 1e29c414b369d4f⋯.gif (47.8 KB, 600x800, 3:4, 1e29c414b369d4f40024146cff….gif)

90f7f9  No.10270[Reply]

I had a great couple of weeks. The login manager is ready for advanced users to play with, and I've done a bunch of other work besides, including plenty of fixes and ui additions, some tag autocomplete speedup, layout improvements for small-screen users, and better en-masse downloader & subscription file management.

Due to the two-week cycle, there is a lot more than usual to test. The release may be a bit late tomorrow.

bdb222  No.10271

>layout improvements for small-screen users

Damn, just today I was thinking about this.

thanks.




YouTube embed. Click thumbnail to play.

7ec229  No.10186[Reply]

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v325/Hydrus.Network.325.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v325/Hydrus.Network.325.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v325/Hydrus.Network.325.-.OS.X.-.App.dmg

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v325/Hydrus.Network.325.-.OS.X.-.Extract.only.tar.gz

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v325/Hydrus.Network.325.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v325.tar.gz

I had a difficult week, but I got some great work done. Save for some final help revisions, the downloader overhaul is complete.

final downloader work

So, I managed to finish 13 of my 15 final jobs in the downloader overhaul. All that remains is a help pass for subscriptions and a better intro to gallery and watcher downloading, which I will fold into normal work over the coming weeks. This has bePost too long. Click here to view the full text.

30 posts and 6 image replies omitted. Click reply to view.

c4d927  No.10259

>>10253

personally for me with how much monster girl shit gets made and put out, and how monsters get lumped in with furry more often than not, so long as the porn is good and not autistic, i'm sure we all know the difference there, I have no problem with it. artist im thinking of specifically right now has no one of fur affinity posting their content outside of it because it's not furry for the most part, so every other place that would normally have their shit just has a small old subsection of it and everywhere else they post themselves is a case of they got sick of posting there.

ill be sure to try it out when it gets put in. but i'm curious, how will hydrus handle captchas? I know this is an issue with some places where they force a captcha regardless of how you log in.


7ec229  No.10261

>>10259

This iteration won't support captchas, but it isn't actually that difficult to do. I supported it several aeons ago when hydrus had a 4chan dumper. Basically, you just fetch a jpg and a challenge key (some text that identifies the problem, and times out after x seconds) from the captcha server, present it to the user, and send back the answer and the same key in the same login form you put in the user/pass. Although some of the tech has moved on since then, so if it is anything like 'select all the road signs', I guess it is a more complicated proposition.

For now, any complicated login will have to rely on the advanced cookies.txt import and just copy cookies from your active logged-in browser (I'll be doing some more work on this in the coming weeks, to make it more compatible).


770b34  No.10262

Thank you based dev.


1a6978  No.10265

Thank you based dev.


7ec229  No.10269

>>10225

Right, I just looked into this, and ffmpeg was giving some funny framerates:

mp4: 0.77 fps 20tbr

webm: 2.5fps 50 tbr

tbr is some kind of fps estimate ffmpeg is pulling from the codec or something. iirc, it stands for:

T: Standards

B: Engineers

R: Are fucked in the head m8, big time

I am not totally sure, but it looks like there is some variable frame rate going on when I look at them in MPC, so I guess it is even more complicated. In any case, I have rejiggered my parsing code to be less trustworthy of these values when they differ like this, and now the videos' frames are counted manually–I now get 7 and 13 respectively, with correct durations.

So these should work better for you in tomorrow's release! As hopefully should the other various low-framerate low-frame-count vids that seem to render all in a rush and then hang on the last frame for ten secs. In v326, turn on help->advanced mode and then right-click these videos and hit reparse files and regen thumbs. They should fix up, at least a bit.

Let me know if you run into any more trouble like this.




File: 3f8a8d22364b1de⋯.jpg (111.13 KB, 400x400, 1:1, 1394958218797.jpg)

d5679d  No.9327[Reply]

Here you can ask questions so that the board is not clogged with small threads.

>>6021 has reached its bump limit, so I made a new thread.

193 posts and 32 image replies omitted. Click reply to view.

406dba  No.10240

>>10239

Yes, like most boorus you can filter tags by preceding them with a hyphen


7df316  No.10241

>>10240

That's fantastic, thanks for the fast response.


e2221b  No.10246

File: c0d6f5a63461bae⋯.png (12.02 KB, 1446x82, 723:41, client_2018-10-13_00-04-06.png)

why are my pixiv downloaders fugged? I'm using v325 and the default pixiv url classes and parsers but not getting much luck. am i missing a step? here's a note:

Page Parser pixiv single file page parser - new layout: Content Parser creator tag: Unable to parse that JSON: No JSON object could be decoded… (Copy note to see full error)

Traceback (most recent call last):

File "include\ClientImportFileSeeds.py", line 1052, in WorkOnURL

all_parse_results = parser.Parse( parsing_context, data )

File "include\ClientParsing.py", line 2098, in Parse

raise e

ParseException: Page Parser pixiv single file page parser - new layout: Content Parser creator tag: Unable to parse that JSON: No JSON object could be decoded


1d6ec9  No.10260

How do you download from pixiv and have it tag each image with the respective pixiv illustration ID? I tried so hard to figure this out on my own but I swear nothing works. I don't understand what to do with the text things on "Hydrus-Presets-and-Scripts", so I just dragged the images into "import downloaders" but even so "all tags" still won't include illustration ID. This is the one thing I need most but I'm not seeing it.


4cd362  No.10268

>>10246

The nips changed things yet again, but should be fixed tomorrow (and hopefully more permanently) thanks to effort from a user, and pixiv tag search is back as well. Let me know if you run into more trouble with it.




File: ab23431121b8563⋯.jpg (96.69 KB, 492x650, 246:325, ab23431121b8563a7bc3b7ac57….jpg)

0675b5  No.5115[Reply]

Gonna start another thread because >>173 is becoming too big for most people.

In the tag manager, whenever multiple tags are selected for multiple files with different tags, if you press the del key, the program will still ask you if you want to delete them all or put them all in the selected files. This does not make sense, if I'm pressing the del key it's obvious that I want them gone.

510 posts and 153 image replies omitted. Click reply to view.

e56036  No.10258

>>10256

It's Turkey. Fuck me if I know why it's blocked honestly, goddamn Wikipedia has been blocked in here for almost 2 years now, I don't think I can understand those people's mindsets. Probably have to do something sexual aspect of the images on danbooru, as far as I know it's banned on Russia and SK too, due to these excuses. Loli is behind a paywall by the way, you need to have a gold account to see it. Only difference between main domain and the clones seems to part before .donmai.us so duplicating the existing Danbooru GUG should work. Thanks for the link, I'll try tomorrow and make another post if I encounter any problems.


4dbbad  No.10263

File: b955b3e4415bffb⋯.png (5.34 KB, 400x195, 80:39, Sans titre.png)

I had to wipe my PTR processing cache recently since client.master.db had fucked itself after processing one of the updates (I was temporarily running on an extremely shaky multi-usb drive setup which probably didn't help), so I've been processing it all back.

I can understand the GUI locking up as this is a heavy operation, but usually for manual PTR syncs it reports progress normally for the first few updates then locks up entirely.

I know it's still processing in the background going by task manager resource values, but not having a proper progress indicator is incredibly grating - been running processing for 3 days and I have no idea where it's at.


4dbbad  No.10264

>>10263

I cancelled and it was at like 2300/5100, for info.

The DB still looked really unstable though, so I just exported my local tags to an archive and fucked off to a backup from a few days ago.

Restoring the tags gave me back the few work I had done between backup and fuckup, but I'm not sure how to make the few files I added as well be re-detected by the client. I guess they're still lurking in the files folders?


4158c4  No.10266

>>10264

Try an orphans check and select to move those files elsewhere rather than delete them. Then you can reimport them


4dbbad  No.10267

>>10266

Didn't know orphan check allowed to move files instead of deleting them, thanks!




File: ea31ab670c8dee7⋯.png (12.55 KB, 340x175, 68:35, mariadb-usa-inc.png)

928501  No.9068[Reply]

ITT: create proposals for making Hydrus more optimized.

Proposal: Why can't Hydrus switch to MariaDB?

If it is faster, then it should be better. The only trouble is having the need to rewrite the queries, which from an SQL standpoint should be a non-issue, right?

List of Databases with Open Source License and Open Source APIs:

SQLite - Currently used in Hydrus, has minimal features

MySQL - A more well-rounded SQL Database with user management

PostgreSQL - An SQL with complex features with less performance

MariaDB - SQL/NoSQL database with heavy optimizations

ElasticSearch - A literal search engine instead of a normal Database

Teradata - IDK

https://www.digitalocean.com/community/tutorials/sqlite-vs-mysql-vs-postgresql-a-comparison-of-relational-database-management-systems

https://www.infoworld.com/article/2611812/mysql/mysql-face-off--mysql-or-mariadb-.html

29 posts and 1 image reply omitted. Click reply to view.

1294e3  No.9886

>>9884

Gelbooru and *.booru.org are hosted in the Netherlands, and they are using "anti-loli laws as an excuse" to force a purge on the admins.


06b08b  No.10077

Do you know how can I convert hydrus db to postgresql? Hydrus db consists of multiple sqlite files, how can I connect all of them?


47c1fb  No.10232


5b84af  No.10247


47c1fb  No.10272




File: 4cd38116eaa8d81⋯.gif (152.57 KB, 1351x412, 1351:412, 4cd38116eaa8d81b9bb50bf0db….gif)

f971b7  No.10221[Reply]

I had a good week, but the login manager work (of which there is a lot!) took up so much time that I only fixed a handful of bugs otherwise. I think it would be better if I spend tomorrow hacking away at more of the login stuff than putting out a small release with some not super-urgent work.

So, I am moving v326 forward to the 17th. Everything else is as normal.



YouTube embed. Click thumbnail to play.

284c42  No.10089[Reply]

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v324/Hydrus.Network.324.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v324/Hydrus.Network.324.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v324/Hydrus.Network.324.-.OS.X.-.App.dmg

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v324/Hydrus.Network.324.-.OS.X.-.Extract.only.tar.gz

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v324/Hydrus.Network.324.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v324.tar.gz

I had a great week. The downloader overhaul is almost done.

pixiv

Just as Pixiv recently moved their art pages to a new phone-friendly, dynamically drawn format, they are now moving their regular artist gallery results to the same system. If your username isn't switched over yet, it likely will be in the coming week.

The change breaks ouPost too long. Click here to view the full text.

40 posts and 9 image replies omitted. Click reply to view.

2f6dcf  No.10188

>>10187

Thanks, I might try running from source. How does this impact the performance?

Oh, it might not be such a big problem after all. It's the import page that automatically opened to show new downloaded images. I must have kept the tab open.

When I open a regular search page for the artist it doesn't show the deleted entries anymore. I guess the import page has a fixed list of entries it displays?


e666e4  No.10189

>>10187

I've had this issue ever since the big UI update/rewrite. Running from source does not help, but it doesn't seem to hurt anything either. I'm running Linux Mint 18.2, which uses Ubuntu 16.04 for its package base.

On an unrelated note, ever since I updated from v304 to v324, my console logs an error on startup. It says, "Could not import lz4." AFAICT, everything works, but I don't know what lz4 does so I don't know if it's just because lz4 code is never called during my normal use of Hydrus.


2f6dcf  No.10190

>>10189

I get that error too. But lz4 is just some obscure type of compression that's not used in any file formats I know, and it doesn't seem to cause problems, so I didn't bother reporting it.


284c42  No.10205

>>10188

Should be about the same, as the frozen exe is just spinning up a python instance and running the same code, afaik. Boot should be a bit faster since it doesn't have to mess around with that bootstrap environment shit.

That's interesting that that page kept the files through the trashed->full delete transition. They should be removed automatically.. That suggests the underlying file context of those pages is 'everything' rather than 'local'. Or maybe they are loading from db subsequently in a non-filtered way. I'll check this, thank you for reporting it!

>>10189

>>10190

Thanks. If you figure out any new/unusual AssertionErrors getting put to your log (not the GTK-critical stuff, which doesn't give me enough info, but a proper python/wx error trace), please let me know. There are still some instabilities in the linux code, but I don't always get a good traceback in my environment.

lz4 isn't a big deal. I've been unable to package it properly for a few months now due to a PyInstaller update. I burned some hours on it and couldn't figure it out, so I will revisit after the python3 update.


2f6dcf  No.10213

>>10205

Okay, I will have a look at running the source then.




File: 1426721772716.png (100.78 KB, 1624x1081, 1624:1081, 1327614072601.png)

7f2c0e  No.471[Reply]


Drag and drop windows with tag rules. Show two windows side by side and one window can be programmed with the rule "ADD tag foo" and the other one has the rule "REMOVE tag foo, ADD tag bar" and you can drag and drop files to them.

Deriving tags from regex of other tags/namespace tags. A file has the tag "filename:big_ugly_name" and we could regex that namespace for another tag.

Tag sets with hotkeys: save a set of tags under a hotkey so it's quick to add them to a file while filtering

Opaque window behind tag list in the corner so it doesn't get hidden by picture background

Option to default certain mime types to be excluded from slideshow and only open externally, will help with videos with odd codecs that don't preview in the slideshow correctly

Option to specify hamming distance in "find similar images", you can't change the option once it's in the filter window and you have to enter the hash manually in the "system:similar to" option
604 posts and 189 image replies omitted. Click reply to view.

ee84fc  No.10199

>>10168

I asked BooruNav, AnimeBoxes and Mignori, and they ALL want us to draft a standard API for them, they recommend a mix of Danbooru and Gelbooru (Sankaku, Moebooru, Derpibooru and MyImouto could be an inspiration as well)


31be42  No.10200

>>10199

Another thing (for those who are open sourced)

https://bitbucket.org/xkero/zenbooru

https://github.com/tjg1/nori

For those who are closed source…

BooruNav - discord.gg/4Sb5YrV

Animeboxes - discord.gg/kYgGJhF

Mignori - discord.gg/zf3SANY


31be42  No.10201


31be42  No.10203

Danbooru/Gelbooru/Sankaku => 150

Shimme => 192

MyImouto/Moebooru => 300

Derpibooru => 50 or 150 or 250


be6069  No.10210

I'd really like some additions to the right click tag > open tag in new search

Most of the time i'm already in a page where i'm searching some tags, so i'd like an option to "open tag in a new search, but also keep tags that are already in the search", as of now i have to readd each tag to the new page

Better yet, an option to also keep the current sorting method instead of opening the new search with the default




File: 06ec0ffde0c293d⋯.png (577 B, 128x128, 1:1, Pyrus.png)

612d3b  No.10129[Reply]

I bet you thought this was a thread made to suck up to Hydrus Dev. HAHA, WRONG!

There's so many things about the Hydrus network that irritate the crap out of me, and it's clear that, as they're deliberate design considerations, they're not going to ever change. Therefore, I want to make my own. Problem is, I haven't really developed shit before, so I just want to know what I'm in for. Maybe I'm mistaken, but I'm under the impression that Hydrus Dev was only starting out when he began creating his as well, so maybe he could offer a few pointers, being already so far into his own project.

Anyway, here's some notes I've been making on how I might do things differently. I imagine Hydrus Dev has already considered a lot of these ideas before, so being more world-weary, and having lost that old shine in his eyes, maybe he could tell me why they're fucking stupid and wouldn't work at all.

4 posts omitted. Click reply to view.

168da2  No.10144

File: 74048034955ad36⋯.png (884.87 KB, 749x499, 749:499, 74048034955ad36ace13079cd4….png)

If you are looking to get into coding something big, this sounds like a great project to go for–it worked for me! I can't speak about your personal situation, but my experience in doing this has been a lot more stress and failure than I expected. The actual code isn't too complicated, but dealing with mistakes and unusual situations (like this week, I had to deal with an issue where searching for '6+girls' on gelbooru with other tags fails because the tag separator is '+', so the combinations needs to be '6%2Bgirls+skirt', which adds a finicky extension to shit I thought was already sorted) is an endless battle. The gulf between 'I have an idea how this could work' and 'hey it works and people like it' is a sea of frustration. I am also not good at collaborating or generally talking to people, which has been a problem on and off.

Figuring out a workflow where I don't go crazy and can keep banging my head against a keyboard every week has been a large part of keeping the project together. I thought it would all be mathematical problem solving, like my programming classes in college, but as I am the only project member, it was instead that and ten other jobs. It takes hours and hours of research, planning, answering feedback, parsing obscure stackexchange articles on conflicting linux .so files, chasing up english-as-second-language bug reports, fighting off overambitious feature requests, and then actually putting text on screen to get stuff going at a v0.1 buggy state. And then, when you have a shaky prototype ready for others to try, you get thanks for your work (which feels great) and multiple bug reports and feature requests (which are all valid and absolutely appreciated but all fundamentally mean more stress for you). Reading your post at >>10128 , I would recommend you try to make a very simple working model and ui first, and then iterate on that. Breadth first rather than depth first. I am not sure what your 'haven't really developed shit before' actually means–if you have never touched an ui library before, I recommend you start extremely basic, like 'Hello World!', and consider symlink-directory-structures once you have a bit more experience with the technical side. If you have never written a line of code before, I would recommend scaling way back and trying tPost too long. Click here to view the full text.


72ae6d  No.10176

File: d40ac46adfae0aa⋯.jpg (62.58 KB, 680x766, 340:383, a64.jpg)

>Doesn't have any experience

>Insists he can do it better


45d2e4  No.10178

>>10137

Starting from scratch and not messing everything up is probably simply harder than just fixing what you don't like in Hydrus.

>>10129

> I just want to know what I'm in for.

Programming? Most people still see nearly endless possibilities to still get better after multiple Ph.D. and 40 years of experience.

Programming teams involving hundreds or thousands of people with 20+ years of experience each do NOT produce perfect code. Nor do companies that put BILLIONS of dollars into the software of their products produce anything nearly flawless.

And so on. That's what you're actually in for. You'll hack something together if you *actually* maintain the motivation for a few thousands of hours (more for you because you don't seem to know how to program and probably don't know the maths and algorithm theory and everything else either), but you can't expect it to be good easily.


ac2774  No.10197

They actually did this https://gitgud.io/Dizmal/borehole


612d3b  No.10244

>>10197

Not me. Why would I come up with a COOL name like "Pyrus" then abandon it?




File: 06ec0ffde0c293d⋯.png (577 B, 128x128, 1:1, Pyrus.png)

899bba  No.10128[Reply]

I bet you thought this was a thread made to suck on Hydrus Dev's cock. HAHA, WRONG!

There's so many things about the Hydrus network that irritate the crap out of me, and it's clear that, as they're deliberate design considerations, they're not going to ever change. Therefore, I want to make my own. Problem is, I haven't really developed shit before, so I just want to know what I'm in for. Maybe I'm mistaken, but I'm under the impression that Hydrus Dev was only starting out when he began creating his as well, so maybe he could offer a few pointers, being already so far into his own project.

Anyway, here's some notes I've been making on how I might do things differently. I imagine Hydrus Dev has already considered a lot of these ideas before, so being more world-weary, and having lost that old shine in his eyes, maybe he could tell me why they're fucking stupid and wouldn't work at all.

Pyrus Database

- Store tags and other filesystem-related metadata in two separate databases for full reversibility. Since I consider reversibility so important, I think it deserves this level of security, to be stored in a more stable database of its own which gets written to less frequently and is therefore less prone to breakage. This secondary database will contain all forms of metadata the source filesystems could possibly support; not just the basics like filenames or creation dates, but also comments music tags, or even multi-purpose "description" fields. Users could also create custom keys such as "Likes" for archived YouTube or Twitter posts. Naturally, this level of data retention may seem redundant to many users, so it wouldn't be mandatory.

- Also store the full source paths of programs, or, optionally, just the final directory name. When certain files are "marked", they may all be stored (or at least symlinked) in a single internal directory with all of their original filenames. This would allow certain flash files, and even games or programs to function normally within the database. Naturally, these directories should be excluded when you start the server up. I'm thinking of a hierarchy like this.


db/.
Post too long. Click here to view the full text.

162f16  No.10175

>>10128

This is the guy from 8/tech/ telling you to show code or GTFO.

> UTF-8 boooo

If you do download images from Pixiv you will see why you are flawed.

> Moar tags

Already on its way to the public

> YouTube and Twitter videos

On the to-do list already

> Symlinks and secondary DB

There are winfags here so it is not for everyone… On the to-do list though.

> No client-server difference

You will regret this once someone implements a 3rd-party client/browser/viewer to work with the upcoming API

> the hideous GTK3 theme

WxPython for Winfag compatibility. Move it to Qt if you have balls.

> No "media viewer", rely on VLC/MPV

On the to-do list

> Allow users to create custom execs for each filetype

Whatever

> You could get a whole lot more complicated in the long-term by introducing a whole scripting language

Read the list of parsing scripts https://github.com/CuddleBear92/Hydrus-Presets-and-Scripts and if you think you can write a language to replace this, take a look under the hood


899bba  No.10194

>>10175

> UTF-8 boooo

I'll forgive you for misinterpreting my rant, but that was about characters being converted to their lower-case forms, not something needless like converting unicode to hex.

Considered the rest though, maybe I'll wait and see how good Hydrus gets.


6efa23  No.10196

>>10194

"Talk is cheap, Show me some code" Linus Torvalds

"Release Early, Release Often, Listen to your customers" Eric S Raymond

"No problem is too big it can't be run away from" Linus Torvalds

"Prototype then polish, Get it working before you optimize it" Eric S Raymond

"Any program is only as good as it is useful" Linus Torvalds

"With enough eyes, All bugs are shallow" Eric S Raymond

"Those that can, do, Those that can't, complain" Linus Torvalds




File: 16d53017153f605⋯.gif (351.59 KB, 320x220, 16:11, 16d53017153f60517bd8babdc8….gif)

703ba0  No.10169[Reply]

I had a difficult week, but I got some great work done. The downloader overhaul is entirely complete but for some help. Tomorrow's release has a variety of unusual new features, such as subscriptions without popups, subscription 'quality' review, improved easy-import download pngs, more download logic controls, improved file system healing, and more.

The release should be as normal tomorrow.

44921d  No.10171

Any chance you'll address my complaints?:

>>10160

>>10161

I can keep patching it myself, but I feel like a relative update interval for high-velocity threads ranging from 5 minutes to 10 seconds is something reasonable (especially because it's not a problem for some imageboards github.com/4chan/4chan-API).


cda5ee  No.10174

Nice. That certainly sounds interesting.

BTW, not sure if you saw it already but Pixiv did recently change the profile pages. Doesn't seem to completely break the downloader, but it seems to result in some failures. Apparently on the first 4 or so downloads it tries.


703ba0  No.10183

>>10171

Thanks, answered in main release thread.

>>10174

Thank you for reporting this–clearly the Nips are at it again. I don't have time to look into it for today's release, but I will check back on Saturday.




YouTube embed. Click thumbnail to play.

9ba0bc  No.9533[Reply]

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v316/Hydrus.Network.316.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v316/Hydrus.Network.316.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v316/Hydrus.Network.316.-.OS.X.-.App.dmg

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v316/Hydrus.Network.316.-.OS.X.-.Extract.only.tar.gz

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v316/Hydrus.Network.316.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v316.tar.gz

I had an excellent week. I was able to cleanly 'do' the big gallery overhaul, resulting in many changes. This is an important release for anyone who does a lot of downloading.

gallery update

So, the meat of the gallery overhaul is complete! Where available, subscriptions, the gallery downloader, and the url downloader will now use the new parsing system to parse 'thumbnail pages'. The new pipelinePost too long. Click here to view the full text.

37 posts and 7 image replies omitted. Click reply to view.

9ba0bc  No.9638

>>9579

>>9580

>>9581

>>9585

>>9586

>>9589

Thank you for this information. The new shortcuts system I am slowly migrating over to does not support double-shortcuts like ZZ, but I hope to eventually move pretty much all events over to it, so remapping arrow keys for movement should be doable in the future. I am drowning in work atm, but as the downloader overhaul comes to a close I should be able to revisit some of these longer term rewrite/cleanup jobs. Let me know how it works for you as I roll it out.


9ba0bc  No.9639

>>9587

I would like to rewrite session loading to make is less resource intensive and do delayed loading like you suggest here.

But yeah also moving to multi- pages should massively reduce the memory burden for downloaders that would otherwise have like 3000 thumbnails attached. Now the db just has to load the file import status list, which is basically just a URL and a couple of numbers per item, rather than all the tags and so on that go with it.

I have some more ideas here to relieve gigantico-clients, but it will have to wait for the next pass on the downloader overhaul.


4bfaae  No.10150

>>9579

>>9580

>>9581

>>9586

>>9589

Let me get this straight. You people LIKE vim? Hell, I use it every so often, but only if I need to make a super-minor change and don't feel like waiting for a graphical text editor to load. Or, more likely, it can't load because I FUBARed my system and need to fix it.

Hell, I've always thought that all of those people talking about and comparing text-only editors were either old farts who worked on mainframes in the 1980s or people just joking around.

I'm not arguing against improvements, mind you. I'm not even arguing about priorities. I'm just confused as all hell.

>>9587

Low-resource mode is a great idea. With the old download system, I've long wished to be able to somehow page out only part of the memory Hydrus was using (yes I know paging doesn't work like that, but let me dream goddammit). At one point I actually bought more RAM so Hydrus worked better, especially when also doing something with my computer that's not Hydrus.


4bfaae  No.10152

>>10150

Shit, I thought this was the new thread.


b0da58  No.10159

>>10150

>I'm just confused as all hell.

Why? There is a lot of people who uses vim as their primary text editor, people really really like it, there is even a game to learn Vim, the community is kinda like Emacs (but I think Emacs users are more dedicated to their text editor), there is a lot of programs that use vim or emacs shortcuts, one example is the Terminal, by the default they use the Emacs shortcuts but you can set it to use Vi shortcuts, I prefer to only use the keyboard so most of my programs use the vi keys (I use them even in my web browser) , I try to do the same with hydrus but it ins't completely possible at least yet.




YouTube embed. Click thumbnail to play.

f48835  No.10032[Reply]

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v323/Hydrus.Network.323.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v323/Hydrus.Network.323.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v323/Hydrus.Network.323.-.OS.X.-.App.dmg

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v323/Hydrus.Network.323.-.OS.X.-.Extract.only.tar.gz

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v323/Hydrus.Network.323.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v323.tar.gz

I had a good week. I finished a decent first version of the downloader easy-import and did some other stuff.

easy import

There is now a simple new dialog under network->downloaders->import downloaders that lets you import new downloaders just by dragging and dropping an encoded png onto it. It tries to do all the semi-complicated comparison and association work to actually build a nPost too long. Click here to view the full text.

27 posts and 12 image replies omitted. Click reply to view.

f48835  No.10085

>>10067

>>10066

Thanks for this update. My guess is opencv is sperging out due to some .so not loading right. I will write a 'image load report mode' in the next week or two that will specify which library is being used as images and gifs load and when failures are occurring.

Are you running from source or a built package?

Anyway, please run this report mode when it comes in and let me know what you see.


f48835  No.10086

>>10068

Yeah, I'd like to write a client api at some point that will let people write scripts to tell the client to do simple stuff like it. It'll just be an http server you can talk to, with some simple authentication.

This will obviously be a big job. It'll be up for vote on the next big poll.


f48835  No.10087

>>10071

>>10069

>>10072

>>10076

Thanks. Although your threads are spiking there, it doesn't look horrific or out of place. My guess is this is ui-bound, particularly since looking at the busy watcher page is making things noticeably more laggy. I will review this code and see if I can optimise and slow the refresh rate based on the number of watchers being tracked.

Although the client uses different threads to do its stuff, python unfortunately cannot use multiple processor cores simultaneously, so it never really is 'truly' multi-threaded. It can use multiple when it dips down to the C++ level (typically when it is doing low-level stuff like heavy math/file processing), but otherwise, it'll typically max out at 12.5% or 25% CPU. My threading code sucks as well (it does a lot of async shit synchronously, just waiting on results), which is a big cause of a lot of ui hang.


0a712b  No.10088

>>10087

One of the things you responded to was a different issue with a subscription.

On the side of reducing load. If possible can you keep the top level things, as I like seeing XXX/XXXX to see how much work is left.

Sometime this week im going to dump the entirety of thread backlogs into a saved session, I have a feeling the watchers are causing quite a bit of ram use even if they aren't visible, it will at least give me a point of comparison once I dump the somewhere around 8000-10000 threads watchers down to around 800 active.


ee828d  No.10093

>>10085

I'm running from source, specifically the AUR package from Arch Linux. Thanks for looking into it, and I'll run in it report mode when the time comes.




File: f7060dfada688c0⋯.gif (45.44 KB, 500x600, 5:6, f7060dfada688c0733299f5bfd….gif)

620020  No.10080[Reply]

I had a great week. As well as some misc work, I churned through about half of the remaining downloader overhaul jobs (things like better ui and options for max jobs per domain), fixed the pixiv downloader, and wrote a prototype twitter username downloader.

The release should be as normal tomorrow.



Delete Post [ ]
[]
Previous [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15]
| Catalog | Nerve Center | Cancer
[ / / / / / / / / / / / / / ] [ dir / agatha2 / animu / arepa / centraca / general / mde / tacos / vg ]