[ home / board list / faq / random / create / bans / search / manage / irc ] [ ]

/hydrus/ - Hydrus Network

Bug reports, feature requests, and other discussion for the hydrus network.

Catalog

See 8chan's new software in development (discuss) (help out)
Infinity Next update (Jan 4 2016)
Name
Email
Subject
Comment *
File
* = required field[▶ Show post options & limits]
Confused? See the FAQ.
Embed
(replaces files and can be used instead)
Options
Password (For file and post deletion.)

Allowed file types:jpg, jpeg, gif, png, webm, mp4, swf, pdf
Max filesize is 8 MB.
Max image dimensions are 10000 x 10000.
You may upload 5 per post.

New user? Start here ---> http://hydrusnetwork.github.io/hydrus/

Currently prioritising: simple IPFS plugin

YouTube embed. Click thumbnail to play.

 No.1714

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v189/Hydrus.Network.189.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v189/Hydrus.Network.189.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v189/Hydrus.Network.189.-.OS.X.-.App.dmg

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v189/Hydrus.Network.189.-.OS.X.-.Extract.only.tar.gz

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v189/Hydrus.Network.189.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v189.tar.gz

I had a pretty good week. I didn't get around to everything I wanted, but I made significant improvements to some formerly big and slow gui and db stuff.

some slow things are faster

I have split the analyze call that usually happens on db update into a lot of smaller jobs that'll be tucked in with the other idle maintenance routines. I shouldn't expect you will ever notice it again. This means that the v188->v189 update step should only take about one second!

I have also improved how the db prepares for a vacuum. It seems to go a little faster now, and if you run Windows, your next vacuum will change something important inside the db that should reduce first-load hdd latency. Let me know if you think your Windows system is running searches faster in the coming week!

Also, I sped up the gui code that adds new thumbnails to an existing set of results, like for file import pages. It was running fast with a small number of thumbnails, but not when there were many thousands. I figured out what was taking so long and rewrote it, and for an example page of 5,000 richly tagged results, I managed to cut ~266ms of processing time down to 3ms! I hope this speeds up some people's large imports in future.

Server and client backup code will also work a hell of a lot faster if you backup onto an existing backup–now, files that have the same size and 'last modified' date (which includes all your regular files and thumbnails, which are not changed by the client) will now be skipped rather than overwritten, saving a lot of time.

Also, I rejigged some maintenance stuff so the client should be able to more quickly respond to a change from idle to not idle when it is in the midst of something big.

full list

- split the big analyze db calls into individual table/index calls and moved them from update code to the normal maintenance routines

- on vacuum, both the client and server dbs will now bump their page size up to 4096 if they are running on windows (server vacuum is triggered by running a backup)

- vacuum should be a slightly faster operation for both the client and server

- boosted the db cache significantly–we'll see if it makes much difference for things.

- the way the selection tags control updates its taglist on increases to its media cache is massively sped up. An update on a 5,000-thumbnail-strong page now typically works in 3ms instead of ~250ms. large import pages should stream new results much more quickly now

- sped up some slow hash calculation code that was lagging a variety of large operations

- some hash caching responsibility is moved about to make it available for the add_media_results comparison, which now typically works in sub-millisecond time (was about 16ms before)

- some sorted media list index recalculation now works faster

- some internal media object hashing is now cached, so sorted list index regeneration is a bit faster

- some medialist file counting is now superfast

- wrote a new pauser object to break big jobs up more conveniently and reduce gui choking

- the repo processing db call now uses this pauser

- some copy and mirror directory functions now use this pauser

- backup and restore code for the client now skips re-copying files if they share the same last modified date and file size

- backup code for the server now skips re-copying files if they share the same last modified date and file size

- http cannotsendrequest and badstatusline errors will now provoke two reattempts before being raised

- socket error 10013 (no access permission, usually due to a firewall) is caught and a nicer error produced

- socket error 10054 (remote host reset connection) is caught, and the connection is reattempted twice before being raised

- the old giphy API is gone, so I have removed giphy

- forced shutdown due to system exit/logoff is handled better

- pubsub-related shutdown exceptions are now caught and silenced

- an unusual shutdown exception is now caught

- fixed a copy subtag menu typo

- cleaned some misc hydrus path code

- tags that begin with a colon (like ':)' ) will now render correctly in the media canvas background

- some misc code cleanup

- dropped flvlib since ffmpeg parses flv metadata better anyway

next week

I still have some things in my github queue, and my private to-do list remains stuffed, but I feel I have improved a lot of broken and slow things over the past month or so and would like to push ahead with something big and new. Thank you all for voting in the poll I set up:

https://poal.me/4bhdd6

The top results right now are:

- IPFS plugin

- suggested tags control

- faster dupe searching

So I will have a bit of a think and get going on those. I don't know how long each will take, but I think I will put about a third of my time into this stuff and see how that works out.

I will start with IPFS , although I still do not know a huge amount about it. I am fairly certain I can upload files into the network, but searching and browsing and hence having a hash to download is something I can't confidently state. Anyway, I will try to get a very simple bridge between hydrus and IPFS working, just to see if it all works.

Otherwise, I will keep working bugs and cleanup.

 No.1715

Great job!

..Though I'm still confused about IPFS integration. Can someone explain all the potential from using it with Hydrus?


 No.1716

>>1715

>Can someone explain all the potential from using it with Hydrus?

I believe the idea is that rather than every single client having to keep a local copy of the public repo, it instead becomes like a big torrent that everyone can just download pieces from as they need.

I think. Is that the idea?


 No.1717

>>1715

Basically IPFS is a peer to peer network for sharing files. Like torrents anyone that chooses to share a particular file helps distribute it to everyone else. Theoretically it should be easier/simpler to dump files onto IPFS to get P2P functionality instead of attempting to re-program the wheel.

What makes IPFS different is that one of their goals is to allow websites to link to IPFS hosted content and have it display directly on their webpage. This is a big deal for people wanting to host, for example, a booru style website but don't have the resources to do so.

The current centralized file repositories would also become obsolete because you would no longer have to depend on them being available to obtain files and you could obtain files from multiple sources.


 No.1718

This thread may be handy for people interested in IPFS

>>>/tech/466819

As well as this article

https://blog.neocities.org/its-time-for-the-permanent-web.html


 No.1719

File: 1452137429914.jpg (37.78 KB, 600x380, 30:19, centralized-decentralized-….jpg)

>>1718

I should have posted this with that.


 No.1721

>>1714

Decent chance we're going to migrate to Next pretty soon.

Might want to post updates on the beta board as well (https://beta.8ch.net/hydrus)


 No.1723

>>1719

Kinda off-topic, but isn't that Distributed concept kinda utopic? I think it'd eventually crumble back into decentralized.


 No.1724

Pressing ctrl+a in the tagging box doesn't select the text I've input

It's not a big issue, but it's a bit annoying because it's done so until recently, and I end up adding tags that don't make any sense.

Also, would being able to sort images by how recently they were archived be possible?

Not at all important, but it would be neat if possible


 No.1725

I'm getting a BSOD with Windows 10, after exiting hydrus. It doesn't happen every time I exit hydrus, but 4 BSODs in the last hour of trying to make it BSOD again.

Kernel_Security_Check_Failure

I'll look at my dmp file and tell you what's in it soon


 No.1726

>>1724

I agree


 No.1729

>>1723

Probably, but it would still be a better decentralized than we have now. I think if IPFS takes off, you'd see those server banks owned by Google and Amazon or whatever go from being the only location of a website, to being giant seedboxes that serve content. They might account for most of the bandwidth still, but the content could be obtained elsewhere if needed.


 No.1736

File: 1452362539582.swf (1.3 MB, 96d3085e7d6adaafbf6a98eb77….swf)

>>1715

>>1716

>>1717

>>1718

>>1719

>>1723

>>1729

I am personally on-the-fence about IPFS specifically but interested in adding more ways for hydrus to put and get data. My aim is to write some new network and db stuff so that hydrus can plug into p2p/distributed stuff more easily, and then write an IPFS plugin for that. If IPFS turns out to be good, then we are off to a running start, and if it turns out to be vapourware, we are still prepared to plug into something else. Moving to distributed communication has been a long-term plan of hydrus, so I am keen to take at least a few steps in this direction.

For a hydrus use-case, I would like to have a checkbox for file repositories, for instance, that would tell the file repo to put all its files on IPFS as well. Then, the file repo could go down (or have very restricted bandwidth allowance), and any client could nonetheless get or prefer to get its files/thumbs from IPFS. Update files (the big whacks of compressed json that hold new mappings and other metadata) would be nice as well, although they can retroactively change and I would need to add some certificate and signing stuff as well. Signing updates has been another long-term plan, as I've also wanted to add client-to-client update sharing.

I am not sure if it is appropriate to put 'expirable' files on IPFS–can you revoke a file, or do they hang around forever?

>>1721

I wrote a little about this in the last update post here >>1704 . I did manage to get into the Board Owner panel in the end, and I actually quite liked it, funnily enough, but since then, my subsequent checks to Next /hydrus/ have been a hanging connection, another 522, a 521, and that bit of 'load balancer/yell at Josh' or whatever it was html. I just clicked it now and got another 521, jej. When it has worked, the options link and the cross to close the new post window don't do anything for me–maybe I just have my browser set up wrong?

You may have noticed the panic threads popping up on the big boards, as well. This all smells of growing drama to me, so I am not terrifically keen to use the new system in any way right now. I wouldn't be surprised if they decide to call the whole thing off, or if we even have another exodus. Or it could all blow over, and this will be something to laugh about in six months.

In any case, I'll be keeping copies of my recent release posts' formatting, so if they do magically get it to work and force us over, I'll recreate the release posts that haven't been migrated.

What fun it is, to be on the internet frontier!

>>1724

Thank you for this report. I will look into it. Shift+Home/End works for me, so if you think of it, you might try that for now.

The way the client understands inbox membership unfortunately makes that difficult, and I also don't store timestamps with any of that information, so I wouldn't be able to sort either. I could write a table to store that info, and retroactively invent some archive times, but I don't really want to bulk out my media result object with more stuff to remember right now. I think you'll have to stick with 'newest/oldest first' and system:age for now. You might have some success if you specifically work on processing your inbox in 'oldest first' order, because you then know that your 'newest first' archived files will be in the order you archived them.

>>1725

I am very sorry you are getting this. I rewrote the exit code last week to fix a system logoff crash, and in order to get it to work, I fudged my response to some system 'pls close' events. It worked fine on Win7, 8.1, Ubuntu, and OS X, but I don't have Win 10 to test it on. As is typical for this sort of thing, the one thing you don't look at is the worst problem.

Is a Win 10 BSOD as serious as for other Windows? Are we talking a full system crash? If so, you should be able to downgrade to v188 this week, and please do–I don't want to be causing you bad problems.

I will give my shutdown code another pass. I will move dialog popups and so on to the places they 'should' be. Please let me know how you get on with v190.


 No.1737

File: 1452368078305.jpg (156.55 KB, 1335x305, 267:61, 1452232207760.jpg)

>>1736

>but since then, my subsequent checks to Next /hydrus/ have been a hanging connection, another 522, a 521, and that bit of 'load balancer/yell at Josh' or whatever it was html. I just clicked it now and got another 521, jej. When it has worked, the options link and the cross to close the new post window don't do anything for me–maybe I just have my browser set up wrong?

Possibly. I was having similar issues when I had localstorage disabled. I think Josh fixed that, though. If you're using Palemoon the website won't work for you.

There have been a few large issues with the server configuration, pic related. It's getting pretty close to being fixed, though.

>You may have noticed the panic threads popping up on the big boards, as well. This all smells of growing drama to me, so I am not terrifically keen to use the new system in any way right now. I wouldn't be surprised if they decide to call the whole thing off, or if we even have another exodus. Or it could all blow over, and this will be something to laugh about in six months.

I don't think either of the first two are likely. The only situation they'd call it off in is if Next just doesn't work, which doesn't seem likely anymore. The issue with an exodus is where people would move to. In the case of 8chan, all of the alternative software is actually worse than Next. There's still going to be a large amount of drama, of course.

Regardless, though, I think just posting on regular 8chan is probably the right choice for now. You'll either be forced to move over to Next or it won't happen at all anyway. It does seem like the former's more like than the latter, at the least.


 No.1742

File: 1452379886046.jpg (108.54 KB, 1173x1536, 391:512, 0289b7d50eb3f0f21330fa2aa6….jpg)

>>1737

Thank you for that update, that makes me feel better. I do use Pale Moon, but I keep hearing that it's no longer in favour. I used to use Opera until they broke that.

Is there any decent non-normie browser out there?

>>1736

I played with IPFS today, and it looks like it can do what we want, including update files. The (semi?) official python api library was a bit janked for me (api.get threw tar exceptions, and api.cat worked but gave unicode strings with raw bytes), so I might just query the daemon through the command line, but I feel good about writing some plugin stuff. I will think about it a bit, and next week I will write a new options panel and the ability to attempt to import an arbitrary multihash from ipfs. After that, I'll look into pushing files from the client into IPFS.


 No.1743

>>1742

>I do use Pale Moon

That would be the issue. I believe that the maintainer of Palemoon hasn't merged something from Firefox's main branch which Next relies on.

>Is there any decent non-normie browser out there?

Not really. You probably have some idea of how shitty browsers are to develop.

Maybe try IceCat or Firefox Developer Edition.

Check out https://wiki.installgentoo.com/index.php/Web_browsers


 No.1747

>>1742

Palemoon is shit, unfortunately. Its prime has long past. Moonchild got it in his head that old people use his browser, instead of power users who would think to search for an alternative to what is already installed on their device. As such he is prioritizing 'never changing anything' over the original vision: provide an optimized build of firefox.

On the flipside, those changes that Moonchild refuses to implement make firefox a lot lighter and faster than it used to be. So… palemoon is a lot worse than firefox. RIP palemoon.

>Is there any decent non-normie browser out there?

Not that I know of. Vanilla firefox is best firefox nowadays. Chromium is best chrome. Why is it so important to use 'non-normie'?


 No.1748

>>1736

>Is a Win 10 BSOD as serious as for other Windows? Are we talking a full system crash?

Yes, and I have more to report. It's not just on exit. There are two consistent variables for every crash:

1) I'm playing a video (could be any player- netflix win10 app, mpv)

2) I'm running hydrus (usually it's starting or exiting or I'm using it in some fashion)

3) BSOD


 No.1751

File: 1452459921283.jpg (28.56 KB, 940x611, 20:13, 1c3a69825e5129f599ab2fa9c5….jpg)

>>1743

>>1747

Thanks. I guess I'll suck it up and move to Firefox, then.

>Why is it so important to use 'non-normie'?

Just that normie programs usually don't prioritise customisability and privacy and apolitical use and the other stuff I personally sperg out about. When I read that the owners of some thing I rely on are launching a new internal initiative for Popular Political Fashion Because It Is [Current Year]™, I get paranoid chills because I have never been popular or fashionable. Nor have I ever wanted to post my achievements to Facebook or ask Cortana where the nearest hip restaurant is. I dislike the way almost all our software is moving in that direction.

I'm probably just being stubborn.

>>1748

That's odd that it is combined with another program. Perhaps having video open launches DirectX or whatever Win 10 uses to draw efficiently to the screen, and that can't handle some out-of-order window creation I am accidentally doing. A full system crash is bad stuff though–it's crazy that an error in my code is something Win 10 can't handle at all.

Anyway, I rewrote the shutdown code today to be a bit more 'proper', and I discovered something that can cause program crash on shutdown in Win 7. This might be the source of your problem as well, so I will make sure to fix it for v190. Setting compatibility mode to Win Vista completely fixes it for me, so if you still have this, you could try that.

But if you are getting the BSOD when hydrus isn't starting or exiting, it might be something else. Could you be getting the BSOD when the autocomplete floating dropdown appears, or maybe when a new popup message arrives?


 No.1752

>>1751

>Just that normie programs usually don't prioritise customisability and privacy and apolitical use and the other stuff I personally sperg out about.

Fortunately, Firefox is fairly customizable and you can have a decent amount of privacy with it (after customization).

>When I read that the owners of some thing I rely on are launching a new internal initiative for Popular Political Fashion Because It Is [Current Year]™, I get paranoid chills

It's rather unfortunate that Mozilla pushes their ideology so hard. Fortunately, the software they make is (mostly) free of that, so you can avoid giving a fuck about Mozilla trying to Get Girls Into Tech™ because the only indication of said attempt is their homepage.

I'd say use their software until its quality is largely impacted by their ideology, since there really isn't anything else to use.

>Nor have I ever wanted to post my achievements to Facebook or ask Cortana where the nearest hip restaurant is. I dislike the way almost all our software is moving in that direction.

With Firefox specifically, you can disable every feature involving anything like that. Of course, they make it more and more obscure to do with every build, but it's still possible.

I don't like the way a lot of software is moving towards Facebook integration™ and "modern design" with voice control etc. either, but that's why I choose to not run any software like that. We're not alone in this dislike, hence why there's still a large crowd developing software that doesn't do that.

The issue, of course, only matters if there are no alternatives; currently, there are.


 No.1753

>>1751

>>1752

http://link-satonaka.tumblr.com/post/113234744119/my-firefox-extensions-for-the-security-and

Basically, request policy, noscript, pureurl, some form of cookie whitelist, some form of referrer control, some form of useragent control. Disable health report. That about covers it, right?

Firefox's UI is, as anon said above, still entirely customizable, despite the whole reskin that users complain "looks just like chrome".

The only thing I'm worried about is that firefox is very soon going to make extension signing mandatory, which will kill off pentadactyl's and maybe requestpolicy's current distribution method. I'm sure the devs will address that, though.

When firefox does eventually go belly up, I'm putting all my eggs into qutebrowser's basket. I think all I need is requestpolicy to be implemented before I would consider switching over

https://github.com/The-Compiler/qutebrowser/issues/28


 No.1754

>>1753

>request policy

>noscript

If I'm not mistaken, uMatrix does what both of these do.

>some form of cookie whitelist

Somewhat related, Self Destructing Cookies might be of interest

>The only thing I'm worried about is that firefox is very soon going to make extension signing mandatory

I can't think of an actual reason for them to do that.

I'm not familiar with pentadactyl. What differs between it and vimperator?

>qutebrowser

I hadn't heard of it before. It looks really nice.

I can't seem to find any mention of an addon system, though. Is there one?


 No.1755

>>1754

>I'm not familiar with pentadactyl. What differs between it and vimperator?

Nevermind. I found the FAQ.


 No.1756

>>1714

on two slightly unrelated notes…

1.i made a irc channel over on rizon (#hydrus) to hang out and discuss about the hydrus network, as for the dev, i'll figure out some way to prove who he is and then give him op/owner

and number 2 is that i kinda rigged the feature poll to how it is, haha. surprised no one picked that up, with one vote being on every choice, i had stopped when ipfs was on 17, and you guys did the rest :)


 No.1757

>>1755

>If I'm not mistaken, uMatrix does what both of these do.

I had not heard of uMatrix, I'll take a look. However, both the requestpolicy and noscript devs are very active and skilled and have been developing their respective extensions for a long time. uMatrix will have to be quite extraordinary to get me to switch. Worth noting that the latest requestpolicy is not on firefox's extension repo, it's RequestPolicyContinued on github.

>Nevermind. I found the FAQ.

The FAQ isn't really helpful. I left vimperator years ago, but I don't think vimperator has cookie management (which is baked into pentadactyl). Otherwise, they're much the same, few minor differences here and there.

However Pentadactyl is updated a lot more than vimperator- there were weekly updates for penta when I left vimp, which had been stale for half a year. Vimp finally released an update or two in 2015, and penta has slowed down significantly, but you're still better off with penta because there have been a LOT of bug fixes since they split, and penta does keep up with the latest firefox, give or take a week.

Something that's not advertised- the pentadactyl devs are actually part of the Suckless team. Suckless are assholes, but I have great respect for their work, and vimperator is dead in the water without the Suckless devs.

Pentadactyl used to have a beta release page where they pushed the newest XPI, but now they seem to have gotten lazy. As far as I know, you need to compile the XPI from source yourself to get the latest version. Get the source from their github page.


 No.1758

>>1754

>I can't seem to find any mention of an addon system, though. Is there one?

Not that I know of. There is talk of a formal plugin system- but I couldn't tell you if that's an idea or if it's already implemented.

There is userscript support

https://github.com/The-Compiler/qutebrowser/blob/master/doc/userscripts.asciidoc

But with the direction qutebrowser is going, I'm hoping to get all my needs baked in.


 No.1759

File: 1452618311254.jpg (475.07 KB, 1280x879, 1280:879, 5afdb719654948f24835b21170….jpg)

>>1756

That's a shame about the poll. I had hoped poal.me would be more robust. Is there a better site you would recommend? Or a better way of canvasing people's opinions?


 No.1780

>>1742

>The (semi?) official python api library was a bit janked for me (api.get threw tar exceptions, and api.cat worked but gave unicode strings with raw bytes), so I might just query the daemon through the command line, but I feel good about writing some plugin stuff.

Maybe worth looking into this:

https://ipfs.io/docs/api/

>Every command usable from the cli is also available through the HTTP API.

This seems like a better way to go about it than dealing with binaries themselves, no need to mess around with file names, paths, etc.



[Return][Go to top][Catalog][Post a Reply]
Delete Post [ ]
[]
[ home / board list / faq / random / create / bans / search / manage / irc ] [ ]