[ home / board list / faq / random / create / bans / search / manage / irc ] [ ]

/scriptcdc/ - Script for Cross Domain Catalog

Making a greasemonkey script for cataloging cross-domain boards

Catalog

8chan Bitcoin address: 1NpQaXqmCBji6gfX8UgaQEmEstvVY7U32C
The next generation of Infinity is here (discussion) (contribute)
A message from @CodeMonkeyZ, 2ch lead developer: "How Hiroyuki Nishimura will sell 4chan data"
Name
Email
Subject
Comment *
File
* = required field[▶ Show post options & limits]
Confused? See the FAQ.
Embed
(replaces files and can be used instead)
Options
Password (For file and post deletion.)

Allowed file types:jpg, jpeg, gif, png, webm, mp4
Max filesize is 8 MB.
Max image dimensions are 10000 x 10000.
You may upload 5 per post.


Issue PERM BAN for any disturbance strictly and immediately in some threads. READ RULES before post.
Tags: #javascript #greasemonkey #2ch
Recommendation: JG, KC/jp/35003, KC/int/26917498

File: 1416838331408.jpg (23.28 KB, 336x357, 16:17, Bugs.jpg)

cd042a No.61

Report bugs and issues here.
Post headlines here and go to discussion thread.(>>62)

Note:
This is one of management threads.
Any subtle disturbance will result in a permanent ban immediately.
Post last edited at

71625f No.84

Format:

[Environment]: Chrome+TamperMonkey and/or FireFox+GreaseMonkey.
[Symptoms]
[Details]
[Workaround]

If the symptoms don't reappear in my environment, I can't fix it. Please describe your issues in details. Details can be written in Japanese.

71625f No.85

File: 1418454358212.png (784.96 KB, 1023x1278, 341:426, 4chan_flag_disappear.png)

[Environment]: Both.
[Symptoms]: I can't see the flags in 4chan/int/ from 8chan or KC.
[Workaround]: Browse /int/ from 4chan.

[Details]: Showing flags are implemented by CSS in 4chan/int/. This script just gather and paste HTMLs to make a catalog, therefore original site's CSS is used for all of them. HTML of 4chan/int/ doesn't contain images of flags. No flags appear in other site's CSS.

71625f No.86

File: 1418454807810.png (667.51 KB, 1022x1278, 511:639, gaps_in_8chan.png)

[Environment]: Both.
[Symptoms]: I mind these gaps.
[Workaround]: Browse from 4chan or KC.

[Details]: These gaps are made by CSS in 8chan.

71625f No.87

File: 1418455300031.png (54.91 KB, 1026x455, 1026:455, leakey.png)

[Environment]: Chrome. (Not evaluated in FireFox)
[Symptoms]: I feel this script is leaky.
[Workaround]: Browse from 4chan and don't read thread of KC.

[Details]: Sometimes 8chan's javascript is leaky. This script unleash you from tight connection between site and its script, so use 4chan to browse 8chan if you feel 8chan is leaky. KC can be an alternative. BTW, I'm investigating leaky issue when I read threads in KC.

71625f No.89

File: 1418456321085.png (86.24 KB, 809x592, 809:592, inconsistency_in_settings.png)

[Environment]: Both.
[Symptoms]: There are inconsistencies of evaluation timings in settings.
[Workaround]: Don't mention.

[Details]: Because I'm bored and tired. I may fix it.

Settings:
Most of them are follows these, but there are exceptions.
Evaluated immediately: Catalog General, Catalog Board Group, UIP tracker for 4chan, Command Line Interface.
Evaluated at making a catalog(you must remake catalog by clicking 'catalog' twice to apply): Catalog Appearance.
Evaluated only start-up.(you must re-load pages to apply): General.

Settings2:
You must click 'ok' or 'apply' to apply. And 'load_default' will apply without clicking 'ok' or 'apply'.

fca118 No.98

[Environment]: Both.
[Symptoms]: The script doesn't work with "https" in 8chan.
[Status]: I'll fix the script.

[Details]: 8chan doesn't accept "https" at first, but now we can use "http". I didn't know the change.

fca118 No.102

>>98
[Status]: Fixed.
[Workaround]: Download v2014.12.14.2 from link of BETA.

cd042a No.114

[Symptoms]: Can't see other domain.
[Environment]: Chrome.(Tampermonkey)
[Workaround]: Fixed. Use v2014.12.21.0 instead of v2014.12.20.0.

[Details]: Tampermonkey makes "twttr:private:requestArticleUrl" access. I don't know why, but it causes error and stops running the script. I modified the script not to cause error by this access.

cd042a No.138

[Symptoms]: Can't update a thread which overs bump limit in native catalog.
[Environment]: Both.
[Workaround]: Fixed. Use v2015.01.16.1.

[Details]: This bug was injected at v2015.01.05.0. (degrade)

cd042a No.146

File: 1422032568789-0.png (12.36 KB, 302x228, 151:114, cache_off.png)

File: 1422032568789-1.png (28.43 KB, 326x561, 326:561, use_json.png)

File: 1422032568789-2.png (34.48 KB, 477x600, 159:200, crawler_target.png)

[Symptoms]: Not stable at scan.
[Environment]: FireFox.
[Workaround]: Read details.

[Details]: Scanning all boards consumes much memory. If it reaches the limit, the scan will stop in most cases, but sometimes crash your browser. To reduce memory consumption, you can choose several approaches.

A. Use other process to scan. Chrome always do this, this is why chrome is stable. In FireFox, you must make another profile to invoke it in other process.
B. Turn off software cache. (Pic.1, reduce 60-200MB) The software cache retains html and json, and you can use it after 2nd search to accelerate. In my environment, it takes to scan 3710 boards as follows.
60 seconds with html from server.
45 seconds with html from cache.
45 seconds with json from server.
17 seconds with json from cache.
C. Change to use json instead of html. (Pic.2, reduce 140MB(=200-60MB)) Unfortunately, html don't have information of thumbnails of webm or swf, so they don't appear in catalog.
D. Reduce num of crawlers. (Pic.3) This increases scanning time, but reduces temporary memory usage at scanning.
E. Reduce number of target boards. (Pic.3) This reduce cache memory consumption, which is mentioned B and C.

cd042a No.164

[Symptoms]: Memory leaks at scan.
[Environment]: Both.
[Workaround]:
1. It seems to be fixed.
2. Use json instead of html. >>146, pic2.
3. I'll support 4chan's native catalog to scan 8chan from 4chan.

[Details]:
Scanning all boards with HTML leaked memory. But very recently it seems to be fixed, so I can't find any memory leaks in scan now. Maybe you don't have to care this.
Last week I found memory leaks in parsing HTML by "DOMParser.parse" command. 8chan's javascript caught it in "MutationObserver" at line 2105 in main.js, and it caused memory leaks. The same matter was happened in >>87, because 8chan's javascript acts as a king who thinks there are no script other than the native. In short, 8chan is not user friendly.
Anyway, all we can do now is to use json (in >>146 pic2) when you feel there are memory leaks. But the json file has much more problems. You can't see thumbnails of webm and youtube, because the file doesn't have any information about those. And more, HW moved some thumbnails to other server, and it's dependent on size of original pic. In short, there are too much inconsistencies in json, so you can't see some thumbnails.
I'll support 4chan's native because of these matters, but it takes time. For now, it seems to be fixed, so the priority is low.



[Return][Go to top][Catalog][Post a Reply]
Delete Post [ ]
[]
[ home / board list / faq / random / create / bans / search / manage / irc ] [ ]