Metal Storm logo
Is webscrapping allowed? For example to get news



Posts: 6   Visited by: 30 users
11.06.2021 - 13:38
ColdRail
Just wondering if is it ok for a discord bot to get the news from the notifications (via pipedream) on my metalstorm account once a day and post them on my channel.
Loading...
11.06.2021 - 21:24
corrupt
With a lowercase c
Admin
It's the Internet. You can do as you please
Don't get me wrong, it's nice of you to ask, and if you overdo it by spamming requests, we will probably ip-ban your bot, but if you're comfortable with handing over your login data to a third party, there's nothing we can effectively do to prevent that. It might be easier and more effective (not to mention less stressful for the site), though, to simply use the news RSS feed and maybe filter that for bands you're interested in than to try and parse MS' horrible HTML with a DOM scraper.
The Metal Storm News account on twitter is nothing else than an RSS parser bot.
----
Loading...
12.06.2021 - 13:56
ColdRail
Ok, good to know. Is also a good idea to parse the XML for RSS Feed, I can use that instead too.

Some websites ban people if they detect bots for example login in at exact same time every day, just wanted to be sure (even if is just one time a day thing)

About the password, last time I checked this website is not HTTPS, thats why I didnt care much about it. I dont feel that my password is safe even when Im login in to this website.
Loading...
12.06.2021 - 15:06
corrupt
With a lowercase c
Admin
Written by ColdRail on 12.06.2021 at 13:56
About the password, last time I checked this website is not HTTPS, thats why I didnt care much about it. I dont feel that my password is safe even when Im login in to this website.

I know dude, you're putting a finger in a big wound, there. And believe me, we're working on it.

But HTTPS alone isn't a factor in your threat model there. That's protection against man-in-the-middle attacks between your browser and the site (and to some extent phishing protection). Not trying to defend that we haven't caught up, there, just saying that's a very unlikely scenario. If, however, you give your login data to a third party, they'll be able to use the site with all your privileges. We have no scheme in place that would allow delegation of specific privileges like oauth. Anything you can do on the site, the third party will be able to do. And that means your account will be banned, if we detect abuse there. That's why an RSS parser might actually be the better idea for you.
----
Loading...
13.06.2021 - 11:49
ColdRail
Wow, nice to know, you dont know how many times I recomended this website and people are too scared because of the browser warning.

I will use the RSS solution
Loading...
14.06.2021 - 14:14
corrupt
With a lowercase c
Admin
Written by ColdRail on 13.06.2021 at 11:49
you dont know how many times I recomended this website and people are too scared because of the browser warning.

Yeah. We're aware of that, too. It is what it is for now. We'll just have to build our user base back up once we got that going
----
Loading...