Page 1 of 1

how do I download a link?

PostPosted: Fri Apr 13, 2012 5:16 pm
by jhb50
In attemting to extract the parameters for an rtmp link in a web resource groovy I found the link http://castamp.com/embed.php?c=puurnld4 (and other c values) could not be opened as a url. ie: String linkHtml = new URL(srcUrl).getText() Explorer also returns a blank page.

I can however download the link using my downloader utility and it contains the necessary rtmp parameter values, so my question is what is the best way to download that link using groovy?

Re: how do I download a link?

PostPosted: Fri Apr 13, 2012 6:20 pm
by zip
the web page is empty but has a source HTML (right click in the browser). It's possible they're expectiv some (browser like) headers, eg User-Agent

Re: how do I download a link?

PostPosted: Fri Apr 13, 2012 10:03 pm
by jhb50
Actually the webpage does load in the browser but is quickly replaced with a blank page as per the html below. So the question remains how do I access that html in groovy when String linkHtml = new URL(srcUrl).getText()
gives me the null page ?

  Code:
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
 
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type" />
<title>CastAMP.com</title>
<script type="text/javascript">
if(self==top) {

window.location = "about:blank";


}
</script>
<noscript><meta http-equiv="refresh" content="0; URL=about:blank"></noscript>
</head>
 
<body>

<div style="text-align:center;width:700px;">

<iframe src="468x60.php" id="header_iframe" frameborder='0' marginheight='0' marginwidth='0' scrolling='no' width='468' height='60'></iframe>
</div>
<div>
<div style="position:relative">
   <div id="ad_overlay" style="width:700px;height:398px;position:absolute;top:0px;z-index:100;background-color:rgba(169, 152, 128, 0.4);display:none;">
      <div id="ad_overlay_countdown" style="700px;position:absolute;top:20px;left:150px;text-align:center;color:#fff;font-size:11px">
This ad will close in <span id="countdownnum"></span> seconds.   <a href="javascript:void(0)" onclick="removeOverlayHTML()"><img style="border:0;z-index:101;display:none" src="http://castamp.com/x.png" alt="x" id="xad" /></a></div>


      <div id="ad_overlay_content" style="width:300px;height:250px;position:absolute;top:35px;left:150px;background:#fff;"></div>
   </div>
      <div id="ad_footer" style="width:700px;height:60px;position:absolute;top:270px;left:6px;background:url();display:none">
      <div id="ad_footer_content" style="position:absolute;width:468px;height:60px;">
<iframe id='footer_iframe' frameborder='0' marginheight='0' marginwidth='0' scrolling='no' width='468' height='60'></iframe>

</div>   
   

   </div>
   <script type="text/javascript" src="swfobject2.js"></script>
      <div id="player">
        <h2>Loading Player...</h2>
   </div>
   <script type='text/javascript'>

  var so = new SWFObject('http://bitcast-r.v1.sjc1.bitgravity.com/malar/Players/player.swf','mpl','700','398','9');
  so.addParam('allowfullscreen','true');
  so.addParam('allowscriptaccess','always');
  so.addParam('wmode','opaque');
//  so.addVariable('skin','http://live.castamp.com/fsjw43.xml');
  so.addVariable('controlbar','over');
  so.addVariable('backcolor','877c6b');
  so.addVariable('dock','false');
  so.addVariable('autostart','true');
  so.addVariable('stretching','uniform');
  so.addVariable('file', 'puurnld4');
  so.addVariable('streamer', 'rtmp://stream.castamp.com:1935/amp');
  //so.addVariable('bufferlength','3');
  so.addVariable('rtmp.tunneling','false');
  so.addVariable('logo.file', 'http://castamp.com/images/castamp.png');
  so.addVariable('logo.hide','false');
  so.addVariable('logo.position','top-right');

  so.addVariable('logo.link','http://castamp.com/home.html');
  so.addVariable('abouttext','CastAMP.com &#45; Free Broadcasting');
  so.addVariable('aboutlink','http://castamp.com/home.html');
  so.write('player');
</script>

Re: how do I download a link?

PostPosted: Fri Apr 13, 2012 10:49 pm
by zip
not sure how they are doing this, but it doesn't seem to be possible. curl and wget fail as well

Re: how do I download a link?

PostPosted: Sat Apr 14, 2012 4:08 am
by jhb50
Using Wireshark here are the requests made by my downloader utility (which downloads the file) and Curl which fails.

Does this give any info that will help solve this problem?

Can anyone write a curl command to duplicate the downloader request?
I've tried but have not succeeded.

  Code:

http:\\www.castamp.com/embed.php?c=puurnld4

CURL
----
GET /embed.php?c=puurnld4 HTTP/1.1
User-Agent: curl/7.23.1 (x86_64-pc-win32 libcurl/7.23.1 OpenSSL/0.9.8r zlib/1.2.5
Host: castamp.com
Accept: */* 

Downloader
----------
GET /embed.php?c=ladytnb HTTP/1.1\r\n
Accept: */*\r\n
Accept-Encoding: gzip, deflate\r\n
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C)\r\n
Host: castamp.com\r\n
Connection: Keep-Alive\r\n
Cookie: __utma=109905047.1530474507.1329370460.1334327012.1334331785.9; __utmz=109905047.1329370460.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)\r\n


Re: how do I download a link?

PostPosted: Sat Apr 14, 2012 3:17 pm
by jhb50
More research finds that
  Code:
curl -X GET "http://castamp.com/embed.php?c=puurnld4"  -H "Accept-Encoding: gzip" -v  > D:\curl.txt


will download a gzip'd version of the file, and I can get the gzip string in groovy with
  Code:
String linkHtml =  new URL(srcUrl).getText(requestProperties:['Accept-Encoding': 'gzip,deflate'])


Can someone tell me how I get groovy to uncompress the gzip format string back to text??

Re: how do I download a link?

PostPosted: Sat Apr 14, 2012 7:17 pm
by zip
I'm using this in Java (you can use it in Groovy too):

  Code:
new GZIPInputStream (is)


You have to pass 'is' which would be the gzipped InputStream, so you have to figure out how to return InputStream from URL rather than text. Looks like then you can call .text on the gzip stream to read its contents like:


  Code:
new GZIPInputStream (is).text


More on Groovy streams is here: http://groovy.codehaus.org/JN2025-Streams

Re: how do I download a link?

PostPosted: Sun Apr 15, 2012 3:17 pm
by jhb50
Thanks..works great

  Code:
import java.util.zip.GZIPInputStream;
String decodedtext = new GZIPInputStream (new URL(address).newInputStream(requestProperties:['Accept-Encoding':
'gzip,deflate'])).text

Re: how do I download a link?

PostPosted: Sun Apr 15, 2012 7:06 pm
by jhb50
Seems I spoke too soon.

The GZIP code
  Code:
String linkHtml = new GZIPInputStream (new URL(srcUrl).newInputStream(requestProperties:['Accept-Encoding': 'gzip,deflate'])).text         

works fine in my groovy in test mode and generates the correct ContentURLContainer
  Code:

ContentURLContainer [fileType=VIDEO, contentUrl=rtmp://50.115.119.42/live/bonanza1.stream swfUrl=http://bitcast-r.v1.sjc1.bitgravity.com/malar/Players/player.swf pageURL=http://castamp.com/home.html playpath=bonanza1.stream, thumbnailUrl=https://sites.google.com/site/hahaflags/en.gif, live=true, expiresOn=Tue May 15 13:12:00 EDT 2012, expiresImmediately=true, cacheKey=http://www.hahasport.com/c-11.html?&estreams=4_http://www.hahasport.com/v-2/18/156/v-395700.html]


however when executed within Serviio the log shows this error:

  Code:
2012-04-15 12:47:00,137 DEBUG [WebResourceParser] Unexpected error during url extractor plugin invocation (HaHaSport) for item Sun 5:00-18:00 (Table 1)World Snooker Championship Qualifying-World Snooker: No signature of method: java.net.URL.newInputStream() is applicable for argument types: (java.util.LinkedHashMap) values: [[requestProperties:[Accept-Encoding:gzip,deflate]]]
Possible solutions: newInputStream(), withInputStream(groovy.lang.Closure)


Why would Serviio process this groovy differently?

Re: how do I download a link?

PostPosted: Sun Apr 15, 2012 11:02 pm
by jhb50
My standalone version of groovy which executes my plug-in correctly is version 1.8.3. What version does Serviio have builtin which fails with my plug-in per my previous post? The failing method in Serviio was implemented in Groovy 1.8.1 so perhaps Serviio is not yet at that level.

Re: how do I download a link?

PostPosted: Mon Apr 16, 2012 4:41 am
by jhb50
Further testing shows that using the Serviio groovy-all.jar for my standalone testing generates the same error, which would appear to prove the Serviio groovy does not have the 1.8.1 release enhancements. The fix would be to include the groovy-1.8.3.jar in the 1.0 release.