Page 1 of 1

How to catch/grabb all links?

Posted: Sun Dec 16, 2018 10:43 am
by Debugger
How to catch all links? (Not to be confused with downloading) I used to have a browser add-on, now I do not know what the add-on is suitable for.

Here from last week's music:
https://soundcloud.com/search/sounds?q= ... =last_week

Re: How to catch/grabb all links?

Posted: Sun Dec 16, 2018 12:16 pm
by therube
See if something here does what you need, Link Bookmarklets?

Re: How to catch/grabb all links?

Posted: Sun Dec 16, 2018 1:08 pm
by Debugger
therube wrote:
Sun Dec 16, 2018 12:16 pm
See if something here does what you need, Link Bookmarklets?
There is nothing here that can catch some or most of the links. The application must process JavaScript (!).
need a proper script to process the links, but do not scroll the page down, which is a damn time-consuming,
I need something like Extract the links on this page

Re: How to catch/grabb all links?

Posted: Sat Apr 20, 2019 10:11 am
by scofficial
Did you tried other available browser extensions?

Re: How to catch/grabb all links?

Posted: Sat Apr 20, 2019 11:59 am
by therube
If you were looking for the song URL, (in a Mozilla browser) you could open Developer Tools -> Network, & filter on .m3u8.

Actually you don't even need to "play" any of them.
So long as you focus (hover) an item, it's URL will show up.

With the URL, you can open it externally, say in MPlayer.

Code: Select all

https://cf-hls-media.sndcdn.com/playlist/NX82UjJgAgGK.128.mp3/playlist.m3u8?Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiKjovL2NmLWhscy1tZWRpYS5zbmRjZG4uY29tL3BsYXlsaXN0L05YODJVakpnQWdHSy4xMjgubXAzL3BsYXlsaXN0Lm0zdTgiLCJDb25kaXRpb24iOnsiRGF0ZUxlc3NUaGFuIjp7IkFXUzpFcG9jaFRpbWUiOjE1NTU3NjE5MjN9fX1dfQ__&Signature=kjXgnwjBwiL35lEmYXQb4mV3RdRVq3MsQk89u1NXjl-34fKiyytYVBroSrFWk-pKvEMUQBu7HE-g-S2Fu40itdsdQDY-XnT4G2wtz3huf2xurRnm1CRQi6yk81y8Xc~V7Fo1XNC-kKYC~oHCHmqjmP4Drtz~nYiPYu5WfWL8QRmg3wlDGxKcP6feIeT8V8QyaIddV8NHDIotu76hQP~sdxXamZdS5p88XBTOvl-talB8ikkzqsYPrIrttaNdxoR3IJwJ0RjDhJ~iM4EtU92kkS4dP--AJzrdYibNtMUmQq1uvszl-S3HZ-IiGOeJXUUahirFv1DQyoLuuA5Kitw3BA__&Key-Pair-Id=APKAJAGZ7VMH2PFPW6UQ
---

Bulk Media Downloader (a sniffer) will not find those links.

Re: How to catch/grabb all links?

Posted: Sat Apr 20, 2019 3:55 pm
by Debugger
therube -
I only need a whole list of source links:

Example:
https://soundcloud.com/user-290759768-1 ... kina-remix
https://soundcloud.com/djparadox79/dj-p ... s-monta-06

Unfortunately, without processing javascript you can not extract the subpage, you have to load it manually.
Well, unless there is an extension that can do it, but there is no such extension anywhere!


Image

Re: How to catch/grabb all links?

Posted: Sun Apr 21, 2019 12:20 pm
by therube
It would seem (in this case) "links" are "generated on the fly" - rather then static, likely have a limited lifespan, & may or may not be accessible to others (such as the link I posted above, is now "dead").

Now maybe you could write some code, the same as the page uses, that iterates through the files shown on the page, generating valid URL to files?

https://www.google.com/search?q=firefox ... +extension
I would be CAREFUL with any of these types of extension found on addons.mozilla.org.
I would be careful, in general, with any extension found on addons.mozilla.org.

Re: How to catch/grabb all links?

Posted: Sun Apr 21, 2019 12:45 pm
by Debugger
therube

First. How wrong you are. The source links are online and play and in no case change. (The material has not been deleted!). Well, unless you mean something else?

Secondly - when it comes to soundcloud.com extensions, they are primarily aimed at downloading files. And that's not what I mean.
But, well - there is an extension to "catching(grabb) all source links", but as I said before, to process the whole page, scroll down the page and then "catch(grabb) all". It's such a fun, but it consumes a lot of time!

Re: How to catch/grabb all links?

Posted: Mon Apr 22, 2019 12:03 pm
by therube
I meant that a link, once "generated", only has a limited lifespan.
The physical file remains, but the link (URL) to that file will differ over time (& also may be "unique" to a particular "user").

Re: How to catch/grabb all links?

Posted: Mon Apr 22, 2019 12:48 pm
by Debugger
therube - Yes, but you must distinguish the type of address, e.g.
Referrer, Source or Data. There are others, but these are the most important ones.

Direct links (DATA) (generated) usually have a limited time (session expiration time, or assigned to your IP). Such files can be downloaded immediately.

------------------------------------
Source.
So they do not change. And I am talking about such links :)
viewforum.php?f=7