From ggallen at slackinc.com Tue Jun 8 11:30:56 2004 From: ggallen at slackinc.com (George Gallen) Date: Tue Jun 8 11:30:30 2004 Subject: [Http-recorder] setting up the proxy.... Message-ID: <6B869BD47339444C958405F559DADCF1074FFD22@smtp.slackinc.com> I setup and ran the proxy.pl program. My PC is a 10.10. address on a network the proxy.pl is running on a server on that network, listening on default 8080 (verified). I set my IE to use the IP address of the server as the proxy server, and port 8080. I verified with tcpdump that the IE is talking to the proxy server. BUT...every URL I put into the IE browser comes back with a DNS error. The server that the proxy is running on has DNS resolution capability. Do I need to reboot after switching to proxy mode in IE? or is there more needed? Or does this only work when the outgoing browser is on the same server as the proxy server? below is the proxy.pl I'm using: #!/usr/bin/perl use HTTP::Proxy; use HTTP::Recorder; my $proxy = HTTP::Proxy->new(); # create a new HTTP::Recorder object my $agent = new HTTP::Recorder; # set the log file (optional) $agent->file("/tmp/myfile"); # set HTTP::Recorder as the agent for the proxy $proxy->agent( $agent ); # start the proxy $proxy->start(); 1; George Gallen Senior Programmer/Analyst Accounting/Data Division ggallen@slackinc.com ph:856.848.1000 Ext 220 SLACK Incorporated - An innovative information, education and management company http://www.slackinc.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.bestpractical.com/pipermail/http-recorder/attachments/20040608/e2c65948/attachment.html From ggallen at slackinc.com Tue Jun 8 11:41:34 2004 From: ggallen at slackinc.com (George Gallen) Date: Tue Jun 8 11:41:50 2004 Subject: [Http-recorder] setting up the proxy.... Message-ID: <6B869BD47339444C958405F559DADCF1074FFD24@smtp.slackinc.com> As a side question to this...Do I need to disable apache running on port 80, prior to running the proxy on port 8080? This is a live system, so I can't "test" it by shutting down apache.... George -----Original Message----- From: George Gallen [mailto:ggallen@slackinc.com] Sent: Tuesday, June 08, 2004 11:31 AM To: http-recorder@lists.bestpractical.com Subject: [Http-recorder] setting up the proxy.... I setup and ran the proxy.pl program. My PC is a 10.10. address on a network the proxy.pl is running on a server on that network, listening on default 8080 (verified). I set my IE to use the IP address of the server as the proxy server, and port 8080. I verified with tcpdump that the IE is talking to the proxy server. BUT...every URL I put into the IE browser comes back with a DNS error. The server that the proxy is running on has DNS resolution capability. Do I need to reboot after switching to proxy mode in IE? or is there more needed? Or does this only work when the outgoing browser is on the same server as the proxy server? below is the proxy.pl I'm using: -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.bestpractical.com/pipermail/http-recorder/attachments/20040608/00b21967/attachment.htm From RGiersig at cpan.org Tue Jun 8 11:45:29 2004 From: RGiersig at cpan.org (Roland Giersig) Date: Tue Jun 8 11:44:59 2004 Subject: [Http-recorder] setting up the proxy.... In-Reply-To: <6B869BD47339444C958405F559DADCF1074FFD22@smtp.slackinc.com> References: <6B869BD47339444C958405F559DADCF1074FFD22@smtp.slackinc.com> Message-ID: <40C5DF19.6020000@cpan.org> George Gallen wrote: > I setup and ran the proxy.pl program. > > My PC is a 10.10. address on a network > the proxy.pl is running on a server on that network, listening on > default 8080 (verified). > > I set my IE to use the IP address of the server as the proxy server, and > port 8080. > > I verified with tcpdump that the IE is talking to the proxy server. > > BUT...every URL I put into the IE browser comes back with a DNS error. Are you sure IE is reporting a DNS error? IE error messages are often misleading. Can you connect to the url in question directly (without proxy)? > The server that the proxy is running on has DNS resolution capability. > > Do I need to reboot after switching to proxy mode in IE? or is there > more needed? > Or does this only work when the outgoing browser is on the same server > as the proxy server? That shouldn't be the case. But I'd try to run proxy.pl on your PC anyway, just to find out if the problem lies at the server. Also, try connecting to the proxy directly via telnet and type in the HTTP GET manually so you can see the exact error returned from the proxy. Hope this helps, Roland From RGiersig at cpan.org Tue Jun 8 11:48:07 2004 From: RGiersig at cpan.org (Roland Giersig) Date: Tue Jun 8 11:47:35 2004 Subject: [Http-recorder] setting up the proxy.... In-Reply-To: <6B869BD47339444C958405F559DADCF1074FFD24@smtp.slackinc.com> References: <6B869BD47339444C958405F559DADCF1074FFD24@smtp.slackinc.com> Message-ID: <40C5DFB7.2010609@cpan.org> George Gallen wrote: > As a side question to this...Do I need to disable apache running on port > 80, prior to > running the proxy on port 8080? This is a live system, so I can't "test" > it by shutting down apache.... No, that shouldn't be necessary. So you are running the proxy on the same server as the website you want to test? Then you could try to use http://localhost/ to browse the website... Roland From ggallen at slackinc.com Tue Jun 8 11:56:53 2004 From: ggallen at slackinc.com (George Gallen) Date: Tue Jun 8 11:56:24 2004 Subject: [Http-recorder] setting up the proxy.... Message-ID: <6B869BD47339444C958405F559DADCF1074FFD25@smtp.slackinc.com> No. I'm running apache on the same server as my PERL interpreter which is running the proxy server. Normally, My PC just goes through a gateway other than the server I have the proxy on (same gateway the server uses as well). So, I'm pointing the PC at the proxy server instead, so I can record some screen entries that are from sites on the internet. Although, I haven't tried installing the modules on my PC, I can try that as well. any clearer? >-----Original Message----- >From: Roland Giersig [mailto:RGiersig@cpan.org] >Sent: Tuesday, June 08, 2004 11:48 AM >To: http-recorder@lists.bestpractical.com >Subject: Re: [Http-recorder] setting up the proxy.... > > >George Gallen wrote: > >> As a side question to this...Do I need to disable apache >running on port >> 80, prior to >> running the proxy on port 8080? This is a live system, so I >can't "test" >> it by shutting down apache.... > >No, that shouldn't be necessary. So you are running the proxy on the >same server as the website you want to test? Then you could try to use >http://localhost/ to browse the website... > >Roland >_______________________________________________ -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.bestpractical.com/pipermail/http-recorder/attachments/20040608/9eadd8c0/attachment.html From RGiersig at cpan.org Tue Jun 8 12:12:12 2004 From: RGiersig at cpan.org (Roland Giersig) Date: Tue Jun 8 12:11:39 2004 Subject: [Http-recorder] setting up the proxy.... In-Reply-To: <6B869BD47339444C958405F559DADCF1074FFD25@smtp.slackinc.com> References: <6B869BD47339444C958405F559DADCF1074FFD25@smtp.slackinc.com> Message-ID: <40C5E55C.7060702@cpan.org> Ah! Is the gateway another proxy? Or are you accessing the internet directly from your PC without configuring a proxy in IE? George Gallen wrote: > No. I'm running apache on the same server as my PERL interpreter > which is running the proxy server. > > Normally, My PC just goes through a gateway other than the server > I have the proxy on (same gateway the server uses as well). > > So, I'm pointing the PC at the proxy server instead, so I can record > some screen entries that are from sites on the internet. > > Although, I haven't tried installing the modules on my PC, I can try > that as well. > > any clearer? > > > > >-----Original Message----- > >From: Roland Giersig [mailto:RGiersig@cpan.org] > >Sent: Tuesday, June 08, 2004 11:48 AM > >To: http-recorder@lists.bestpractical.com > >Subject: Re: [Http-recorder] setting up the proxy.... > > > > > >George Gallen wrote: > > > >> As a side question to this...Do I need to disable apache > >running on port > >> 80, prior to > >> running the proxy on port 8080? This is a live system, so I > >can't "test" > >> it by shutting down apache.... > > > >No, that shouldn't be necessary. So you are running the proxy on the > >same server as the website you want to test? Then you could try to use > >http://localhost/ to browse the website... > > > >Roland > >_______________________________________________ > > > ------------------------------------------------------------------------ > > _______________________________________________ > Http-recorder mailing list > Http-recorder@lists.bestpractical.com > http://lists.bestpractical.com/cgi-bin/mailman/listinfo/http-recorder From ggallen at slackinc.com Tue Jun 8 12:20:50 2004 From: ggallen at slackinc.com (George Gallen) Date: Tue Jun 8 12:20:18 2004 Subject: [Http-recorder] setting up the proxy.... Message-ID: <6B869BD47339444C958405F559DADCF1074FFD28@smtp.slackinc.com> >-----Original Message----- >From: Roland Giersig [mailto:RGiersig@cpan.org] >Sent: Tuesday, June 08, 2004 11:45 AM >To: http-recorder@lists.bestpractical.com >Subject: Re: [Http-recorder] setting up the proxy.... > > >Also, try connecting to the proxy directly via telnet and type in the >HTTP GET manually so you can see the exact error returned from >the proxy. > This is strange. If I telent to my port 80 (apache), and type in a GET response, I get back what I'd expect, in this case I put in a bogus page, and got back a page not found error from apache. If I telnet to my port 8080 (proxy) it drops the connection, however, if I telnet to port 8080 from the server (localhost), and put in a GET response it works. nothing gets put to the log file..but I'll work with that later. I don't understand why I'm able to telnet to port 80 but not 8080 from the PC (outside the server), but can telnet to either port from inside the server. Does the proxy use any form of hosts.allow/deny that I need to set? or will it only allow localhost to connect to itself (like java)? Thanks George >Hope this helps, > >Roland >_______________________________________________ >Http-recorder mailing list >Http-recorder@lists.bestpractical.com >http://lists.bestpractical.com/cgi-bin/mailman/listinfo/http-recorder > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.bestpractical.com/pipermail/http-recorder/attachments/20040608/82224a1a/attachment.htm From RGiersig at cpan.org Tue Jun 8 12:29:35 2004 From: RGiersig at cpan.org (Roland Giersig) Date: Tue Jun 8 12:29:02 2004 Subject: [Http-recorder] setting up the proxy.... In-Reply-To: <6B869BD47339444C958405F559DADCF1074FFD28@smtp.slackinc.com> References: <6B869BD47339444C958405F559DADCF1074FFD28@smtp.slackinc.com> Message-ID: <40C5E96F.10205@cpan.org> George Gallen wrote: > >-----Original Message----- > >From: Roland Giersig [mailto:RGiersig@cpan.org] > >Sent: Tuesday, June 08, 2004 11:45 AM > >To: http-recorder@lists.bestpractical.com > >Subject: Re: [Http-recorder] setting up the proxy.... > > > > > >Also, try connecting to the proxy directly via telnet and type in the > >HTTP GET manually so you can see the exact error returned from > >the proxy. > > > > This is strange. > > If I telent to my port 80 (apache), and type in a GET response, I get back > what I'd expect, in this case I put in a bogus page, and got back a page > not found error from apache. > > If I telnet to my port 8080 (proxy) it drops the connection, however, if I > telnet to port 8080 from the server (localhost), and put in a GET > response > it works. nothing gets put to the log file..but I'll work with that > later. > > I don't understand why I'm able to telnet to port 80 but not 8080 from the > PC (outside the server), but can telnet to either port from inside the > server. > > Does the proxy use any form of hosts.allow/deny that I need to set? or > will it only allow localhost to connect to itself (like java)? It could be that the proxy only binds the socket for 8080 for localhost, not for general IP adresses (but I find that highly improbable). What is the exact error? Can telnet establish a connection, but it gets closed immediately? This would point to some sort of access control. Or, if the connection cannot be established per se, then the port isn't visible from outside the server, i.e. the socket is bound to accept only from localhost. Roland From ggallen at slackinc.com Tue Jun 8 12:46:47 2004 From: ggallen at slackinc.com (George Gallen) Date: Tue Jun 8 12:46:50 2004 Subject: [Http-recorder] setting up the proxy.... Message-ID: <6B869BD47339444C958405F559DADCF1074FFD29@smtp.slackinc.com> to your first question, no the gateway isn't a proxy as well. As for access control... >From the PC side, it looks like it's being dropped immediately, but from the server side (tcpdump), there is negotiation, so I know it's reaching the server. and when the proxy server isn't running, there is the same negotiation, which is also the same as when I try to telnet to another port which doesn't have a listener. Possibly it's access control on the system side, but it shouldn't. I'll have to try to set it up on the PC side now and see what I get. Thanks George >-----Original Message----- >From: Roland Giersig [mailto:RGiersig@cpan.org] >Sent: Tuesday, June 08, 2004 12:30 PM >To: http-recorder@lists.bestpractical.com >Subject: Re: [Http-recorder] setting up the proxy.... > > >It could be that the proxy only binds the socket for 8080 for >localhost, >not for general IP adresses (but I find that highly >improbable). What is >the exact error? Can telnet establish a connection, but it gets closed >immediately? This would point to some sort of access control. >Or, if the >connection cannot be established per se, then the port isn't visible >from outside the server, i.e. the socket is bound to accept only from >localhost. > >Roland > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.bestpractical.com/pipermail/http-recorder/attachments/20040608/c91ae145/attachment.html From ggallen at slackinc.com Tue Jun 8 16:00:04 2004 From: ggallen at slackinc.com (George Gallen) Date: Tue Jun 8 15:59:31 2004 Subject: [Http-recorder] setting up the proxy.... Message-ID: <6B869BD47339444C958405F559DADCF1074FFD32@smtp.slackinc.com> Well, it was the proxy module that was limiting it to the localhost access only. If I added in $proxy->host(""); after the line in the proxy sample program, it will allow any host to proxy (NOT A GOOD SECURITY OPTION), but it is ok for testing :) my $proxy = HTTP::Proxy->new(); $proxy->host(""); George >From: Roland Giersig [ mailto:RGiersig@cpan.org] >Sent: Tuesday, June 08, 2004 12:30 PM >To: http-recorder@lists.bestpractical.com >Subject: Re: [Http-recorder] setting up the proxy.... > > >It could be that the proxy only binds the socket for 8080 for >localhost, >not for general IP adresses (but I find that highly >improbable). What is >the exact error? Can telnet establish a connection, but it gets closed >immediately? This would point to some sort of access control. >Or, if the >connection cannot be established per se, then the port isn't visible >from outside the server, i.e. the socket is bound to accept only from >localhost. > >Roland > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.bestpractical.com/pipermail/http-recorder/attachments/20040608/799d8555/attachment.htm From aidan.kehoe at phorest.ie Wed Jun 9 11:03:00 2004 From: aidan.kehoe at phorest.ie (Aidan Kehoe) Date: Fri Jun 11 08:22:29 2004 Subject: [HTTP-Recorder] Issue with some forms Message-ID: <40C726A4.3050202@phorest.ie> Hi, HTTP::Recorder wasn't stripping all of its extra POST variables before passing request to the server, and this broke some of my scripts. There's a patch to fix that attached. Bye, -- Aidan Kehoe Technical Officer nDevor Systems Ltd Email: aidan.kehoe@phorest.ie Mobile: +353 86 3874935 Phone: +353 1 8747800 (GMT office hours) Physical: 69 Middle Abbey Street, Dublin 1, Ireland Web: http://www.phorest.com/ -------------- next part -------------- --- Recorder.pm.orig Wed Jun 9 15:57:15 2004 +++ Recorder.pm Wed Jun 9 15:59:31 2004 @@ -177,7 +177,6 @@ my $content = shift; # get rid of the stuff we added - $content =~ s/($prefix-form-(\d+)-)//g; $content =~ s/$prefix-(.*?)\?(.*?)&//g; $content =~ s/$prefix-(.*?)&//g; $content =~ s/$prefix-(.*?)$//g; From Roland at Giersig.org Tue Jun 8 12:28:59 2004 From: Roland at Giersig.org (Roland Giersig) Date: Fri Jun 11 08:24:14 2004 Subject: [Http-recorder] setting up the proxy.... In-Reply-To: <6B869BD47339444C958405F559DADCF1074FFD28@smtp.slackinc.com> References: <6B869BD47339444C958405F559DADCF1074FFD28@smtp.slackinc.com> Message-ID: <40C5E94B.1040702@Giersig.org> George Gallen wrote: > >-----Original Message----- > >From: Roland Giersig [mailto:RGiersig@cpan.org] > >Sent: Tuesday, June 08, 2004 11:45 AM > >To: http-recorder@lists.bestpractical.com > >Subject: Re: [Http-recorder] setting up the proxy.... > > > > > >Also, try connecting to the proxy directly via telnet and type in the > >HTTP GET manually so you can see the exact error returned from > >the proxy. > > > > This is strange. > > If I telent to my port 80 (apache), and type in a GET response, I get back > what I'd expect, in this case I put in a bogus page, and got back a page > not found error from apache. > > If I telnet to my port 8080 (proxy) it drops the connection, however, if I > telnet to port 8080 from the server (localhost), and put in a GET > response > it works. nothing gets put to the log file..but I'll work with that > later. > > I don't understand why I'm able to telnet to port 80 but not 8080 from the > PC (outside the server), but can telnet to either port from inside the > server. > > Does the proxy use any form of hosts.allow/deny that I need to set? or > will it only allow localhost to connect to itself (like java)? It could be that the proxy only binds the socket for 8080 for localhost, not for general IP adresses (but I find that highly improbable). What is the exact error? Can telnet establish a connection, but it gets closed immediately? This would point to some sort of access control. Or, if the connection cannot be established per se, then the port isn't visible from outside the server, i.e. the socket is bound to accept only from localhost. Roland From lambretta at inet.uni2.dk Fri Jun 11 16:01:58 2004 From: lambretta at inet.uni2.dk (allan juul) Date: Fri Jun 11 16:01:26 2004 Subject: [HTTP-Recorder] how to set the proxy Message-ID: <40CA0FB6.3050607@inet.uni2.dk> hi having read the article at http://www.perl.com/pub/a/2004/06/04/recorder.html i got stuck pretty early, probably because i dont fully understand what's going on. the docs says : "Then, tell your web browser to use this proxy, and the script will be recorded in the specified file." my question is, exactly how do i tell my webbrowser to use this proxy ? ./allan From marc at questright.com Fri Jun 11 22:37:17 2004 From: marc at questright.com (Marc Mims) Date: Fri Jun 11 22:36:49 2004 Subject: [HTTP-Recorder] Article / code mismatch? Message-ID: <20040612023717.GB5197@questright.com> I must be missing something. I read the "Web Testing with HTTP::Recorder article at: http://www.perl.com/pub/a/2004/06/04/recorder.html I created a script directly from the source in the article: #!/usr/bin/perl use HTTP::Proxy; use HTTP::Recorder; my $proxy = HTTP::Proxy->new(); # create a new HTTP::Recorder object my $agent = new HTTP::Recorder; # set the log file (optional) $agent->file("/tmp/myfile"); # set HTTP::Recorder as the agent for the proxy $proxy->agent( $agent ); # start the proxy $proxy->start(); 1; Running it generates the error message: Can't locate object method "file" via package "HTTP::Recorder" at ./http-recorder line 12. I worked around that by providing a the file option to the HTTP::Recorder constructor: my $agent = new HTTP::Recorder(file => "/tmp/myfile"); That works, and I get the expected output in the log file. But I've had no success attempting to access the HTTP::Recorder Control Panel: 500 Can't connect to http-recorder:80 (Bad hostname 'http-recorder') The article says: See Configuration Options for more information about setting up the control URL. However, the link to Configuration options is a dead link: http://www.perl.com/pub/a/2004/06/04/recorder.html#config Surely I'm missing something. I've seen messages on the list containing the original script from the article with every indication that it is working. What might I be missing? -Marc From ivan at pechorin.com Sat Jun 12 04:11:06 2004 From: ivan at pechorin.com (Ivan Pechorin) Date: Sat Jun 12 04:10:45 2004 Subject: [HTTP-Recorder] Article / code mismatch? In-Reply-To: <20040612023717.GB5197@questright.com> References: <20040612023717.GB5197@questright.com> Message-ID: <1087027865.1869.9.camel@localhost.localdomain> On Sat, 2004-06-12 at 06:37, Marc Mims wrote: > The article says: See Configuration Options for more information > about setting up the control URL. However, the link to Configuration > options is a dead link: > http://www.perl.com/pub/a/2004/06/04/recorder.html#config Try one-page version of the article: http://www.perl.com/lpt/a/2004/06/04/recorder.html link to Configuration should work with it: http://www.perl.com/lpt/a/2004/06/04/recorder.html#config > Surely I'm missing something. I've seen messages on the list > containing the original script from the article with every indication > that it is working. What might I be missing? I think, you are using version 0.02 of HTTP::Recorder and the article describes version 0.03_01, marked as "Developer release" on CPAN. You should install it manually: http://search.cpan.org/CPAN/authors/id/L/LE/LEIRA/HTTP-Recorder-0.03_01.tar.gz -- Ivan Pechorin From ivan at pechorin.com Sat Jun 12 04:51:50 2004 From: ivan at pechorin.com (Ivan Pechorin) Date: Sat Jun 12 04:51:18 2004 Subject: [HTTP-Recorder] how to set the proxy In-Reply-To: <40CA0FB6.3050607@inet.uni2.dk> References: <40CA0FB6.3050607@inet.uni2.dk> Message-ID: <1087030309.1869.65.camel@localhost.localdomain> On Sat, 2004-06-12 at 00:01, allan juul wrote: > the docs says : > "Then, tell your web browser to use this proxy, and the script will be > recorded in the specified file." > > my question is, exactly how do i tell my webbrowser to use this proxy ? Allan, how do you usually tell you web browser to use http proxy? Proxy address is 127.0.0.1 Proxy port, used by default, is 8080 P.S. There are some tips on setting up proxy in the Internet Explorer and Mozilla at http://www.freeproxy.ru/en/free_proxy/howuse.htm -- Ivan Pechorin From ivan at pechorin.com Sat Jun 12 05:23:39 2004 From: ivan at pechorin.com (Ivan Pechorin) Date: Sat Jun 12 05:23:08 2004 Subject: [HTTP-Recorder] Issue with some forms In-Reply-To: <40C726A4.3050202@phorest.ie> References: <40C726A4.3050202@phorest.ie> Message-ID: <1087032218.1869.77.camel@localhost.localdomain> Hi! On Wed, 2004-06-09 at 19:03, Aidan Kehoe wrote: > HTTP::Recorder wasn't stripping all of its extra POST variables before passing > request to the server, and this broke some of my scripts. There's a patch to > fix that attached. Is it a patch against version 0.02 of HTTP::Recorder? Seems like this bug is already fixed in 0.03_01. Bye, -- Ivan Pechorin From leira at mit.edu Sat Jun 12 08:56:33 2004 From: leira at mit.edu (Linda L. Julien) Date: Sat Jun 12 08:56:52 2004 Subject: [HTTP-Recorder] Article / code mismatch? In-Reply-To: <20040612023717.GB5197@questright.com> (message from Marc Mims on Fri, 11 Jun 2004 19:37:17 -0700) References: <20040612023717.GB5197@questright.com> Message-ID: <200406121256.IAA20503@small-gods.mit.edu> Sorry about that...the latest version of the code, with the features described in the article, was released as a developer release, so you may not get it from CPAN by default. The version you want is 0.03_01. I'll bumpt the version number and do a "real" release as soon as I get the chance. Linda From lambretta at inet.uni2.dk Sat Jun 12 16:06:12 2004 From: lambretta at inet.uni2.dk (allan juul) Date: Sat Jun 12 16:05:35 2004 Subject: [HTTP-Recorder] how to set the proxy In-Reply-To: <1087030309.1869.65.camel@localhost.localdomain> References: <40CA0FB6.3050607@inet.uni2.dk> <1087030309.1869.65.camel@localhost.localdomain> Message-ID: <40CB6234.5070207@inet.uni2.dk> thanks it's quite obvious now ;) ./Allan Ivan Pechorin wrote: > On Sat, 2004-06-12 at 00:01, allan juul wrote: > > >>the docs says : >>"Then, tell your web browser to use this proxy, and the script will be >>recorded in the specified file." >> >>my question is, exactly how do i tell my webbrowser to use this proxy ? > > > Allan, how do you usually tell you web browser to use http proxy? > > Proxy address is 127.0.0.1 > Proxy port, used by default, is 8080 > > P.S. There are some tips on setting up proxy in the Internet Explorer > and Mozilla at http://www.freeproxy.ru/en/free_proxy/howuse.htm > From magnus.hacker at nfs.se Mon Jun 14 06:08:02 2004 From: magnus.hacker at nfs.se (Magnus Hacker) Date: Mon Jun 14 06:05:44 2004 Subject: [HTTP-Recorder] Slow access Message-ID: <40CD7902.5040009@nfs.se> Hi, I'm experiencing that HTTP::Recorder is running very slowly (could be HTTP::Proxy also of course). This is a log from accessing www.perl.org: [Mon Jun 14 11:55:18 2004] Forked child process -1152 [Mon Jun 14 11:55:27 2004] (-1152) Request: GET http://www.perl.org/ [Mon Jun 14 11:55:29 2004] (-1152) Response: 200 OK [Mon Jun 14 11:55:30 2004] (-1152) Request: GET http://www.perl.org/css/perl.cs [Mon Jun 14 11:55:31 2004] Forked child process -460 [Mon Jun 14 11:55:31 2004] (-1152) Response: 200 OK [Mon Jun 14 11:55:45 2004] (-460) Request: GET http://www.perl.org/css/perl.css rec-url=%2Fcss%2Fperl.css&rec-action=norecord [Mon Jun 14 11:55:46 2004] (-460) Response: 200 OK [Mon Jun 14 11:55:46 2004] (-1152) Request: GET http://www.perl.org/favicon.ico rec-url=%2Ffavicon.ico&rec-action=norecord [Mon Jun 14 11:55:46 2004] (-460) Request: GET http://www.perl.org/images/butto s/download_perl1.gif [Mon Jun 14 11:55:47 2004] (-460) Response: 200 OK [Mon Jun 14 11:55:48 2004] (-1152) Response: 200 OK [Mon Jun 14 11:55:48 2004] Forked child process -2752 [Mon Jun 14 11:55:49 2004] (-2752) Request: GET http://www.perl.org/simages/lca el.gif [Mon Jun 14 11:55:50 2004] (-2752) Response: 200 OK [Mon Jun 14 11:55:50 2004] Forked child process -1364 [Mon Jun 14 11:56:56 2004] (-1364) Request: GET http://www.perl.org/images/frie ds/bizrate_logo.png [Mon Jun 14 11:56:56 2004] (-1364) Response: 200 OK [Mon Jun 14 12:00:59 2004] (-2752) Served 1 requests [Mon Jun 14 12:00:59 2004] (-460) Served 2 requests [Mon Jun 14 12:00:59 2004] (-1152) Served 3 requests [Mon Jun 14 12:01:00 2004] Reaped child process -1152 [Mon Jun 14 12:01:00 2004] Remaining kids: -460 -2752 -1364 [Mon Jun 14 12:01:00 2004] Reaped child process -460 [Mon Jun 14 12:01:00 2004] Remaining kids: -2752 -1364 [Mon Jun 14 12:01:00 2004] Reaped child process -2752 [Mon Jun 14 12:01:00 2004] Remaining kids: -1364 I'm using Mozilla 1.8a1, Perl 5.8.3 (ActivePerl) with HTTP::Recorder 0.3.1 and HTTP::Proxy 0.13. PPM wouldn't find any of Recorder or Proxy so I installed them using the package .tar.gz instead. I'm running on Windows XP and can't see any ununusal paging activity or high cpu usage. thanks /Hacker From jw at mailsw.com Mon Jun 14 14:02:13 2004 From: jw at mailsw.com (Jonathan Wilson) Date: Mon Jun 14 14:01:55 2004 Subject: [HTTP-Recorder] http-recorder adding ",1" to all input fields Message-ID: <200406141302.13905.jw@mailsw.com> Hello, I'm attempting to use HTTP::Recorder to make some WWW:Mechanize scripts for testing a web site. I have it mostly working but I'm having to problems: 1. I can't access the control panel. I've tried the default URL and also tried changing it: control => "http://reccontrol/" But all I get is bad host name responses. However, I can live with that as I don't absolutely have to have the control panel. 2. What I do have to have is the ability to submit a form with some information. However when I do that a strange thing is happening: some part or the proxy is adding ",1" to everything. For example I have a page on the site where I can order by catalog number -- all it wants it the number in the first field and quantity in the second. So I would normally put in something like this: BLAS 25 (and yes, there is nothing wrong with the web site I'm testing, I double-checked that) What happens when I use the proxy is that I get a response back from the web site that the catalog number cannot be found, and with the fields changed to: BLAS,1 25,1 I have tried this on a couple browsers (Konqueror, Mozilla 1.6) and it works without the proxy but alway fail when the proxy is in use. I am able to follow links with the proxy just fine, and it appears to be recording ok. My script to run the proxy is: -------------------BEGIN SCRIPT--------------------- #!/usr/bin/perl -w #This is HTTP::Recorder running as a daemon with HTTP::Proxy use HTTP::Proxy; use HTTP::Recorder; my $proxy = HTTP::Proxy->new(); # set HTTP::Recorder as the agent my $agent = HTTP::Recorder->new( file => "/tmp/http-recorder.log" ); # showwindow => 1 # spare futile attempts at accessing control panel # control => "http://reccontrol/", # create a new HTTP::Recorder object #my $agent = new HTTP::Recorder; # set the log file (optional) #$agent->file("/tmp/http-recorder.log"); # set HTTP::Recorder as the agent for the proxy $proxy->agent( $agent ); # you may need to set the host $proxy->host( "jw" ); # start the proxy $proxy->start(); 1; --------------------------END SCRIPT------------------------- One other little thing I noticed: you cannot set the log file or anything else in this manner: $agent->file("/tmp/http-recorder.log"); It only works as file => "/tmp/http-recorder.log" inside HTTP::Recorder->new(); (Might not matter but I just thought I'd point it out as one other thing that wasn't working right. Using perl v5.8.4 from CPAN on Linux 2.4.21 If anyone can shed some light on the input field problem I would really appreciate it. Thanks! JW -- ---------------------------------------------------- Jonathan Wilson Cedar Creek Software http://www.cedarcreeksoftware.com From ivan at pechorin.com Mon Jun 14 15:41:16 2004 From: ivan at pechorin.com (Ivan Pechorin) Date: Mon Jun 14 15:40:48 2004 Subject: [HTTP-Recorder] http-recorder adding ",1" to all input fields In-Reply-To: <200406141302.13905.jw@mailsw.com> References: <200406141302.13905.jw@mailsw.com> Message-ID: <1087242075.16595.14.camel@localhost.localdomain> On Mon, 2004-06-14 at 22:02, Jonathan Wilson wrote: > 1. I can't access the control panel. > I've tried the default URL and also tried changing it: > control => "http://reccontrol/" > But all I get is bad host name responses. Jonathan, try to specify only host part of the URL: control => "reccontrol" > One other little thing I noticed: you cannot set the log file or anything else > in this manner: $agent->file("/tmp/http-recorder.log"); > It only works as file => "/tmp/http-recorder.log" inside > HTTP::Recorder->new(); > (Might not matter but I just thought I'd point it out as one other thing that > wasn't working right. It seems like you are using version 0.02 of HTTP::Recorder, which is installed from CPAN by default. There is a newer version of HTTP::Recorder, marked as "developer release": http://search.cpan.org/CPAN/authors/id/L/LE/LEIRA/HTTP-Recorder-0.03_01.tar.gz It incorporates some fixes. -- Ivan Pechorin From jw at mailsw.com Mon Jun 14 16:30:47 2004 From: jw at mailsw.com (Jonathan Wilson) Date: Mon Jun 14 16:30:18 2004 Subject: [HTTP-Recorder] http-recorder adding ",1" to all input fields In-Reply-To: <1087242075.16595.14.camel@localhost.localdomain> References: <200406141302.13905.jw@mailsw.com> <1087242075.16595.14.camel@localhost.localdomain> Message-ID: <200406141530.47481.jw@mailsw.com> IP >There is a newer version of HTTP::Recorder, marked as "developer Thanks, I am using 0.03_01 now, and -- most importantly -- the ",1" problem is fixed. Also I can now access the control panel now (with either the default url or a modified onw), though it doesn't work - when I put a url into the Goto page field I get: An error occurred while loading http://http-recorder/: Connection to host http-recorder is broken Doesn't matter too much , though it might be handy for SSL testing some time. Thank you "very muchly" for your help :-) -- ---------------------------------------------------- Jonathan Wilson Cedar Creek Software http://www.cedarcreeksoftware.com From jw at mailsw.com Mon Jun 14 16:39:50 2004 From: jw at mailsw.com (Jonathan Wilson) Date: Mon Jun 14 16:39:21 2004 Subject: [HTTP-Recorder] [solved] http-recorder adding ", 1" to all input fields In-Reply-To: <200406141530.47481.jw@mailsw.com> References: <200406141302.13905.jw@mailsw.com> <1087242075.16595.14.camel@localhost.localdomain> <200406141530.47481.jw@mailsw.com> Message-ID: <200406141539.50408.jw@mailsw.com> JW >Also I can now access the control panel now (with either the default url or a JW >modified one), though it doesn't work - when I put a url into the Goto page JW >field I get: Actually that works too, sorry. Thanks again. -- ---------------------------------------------------- Jonathan Wilson Cedar Creek Software http://www.cedarcreeksoftware.com From chris.mcmahon at verint.com Mon Jun 14 17:21:09 2004 From: chris.mcmahon at verint.com (McMahon, Chris) Date: Mon Jun 14 17:20:30 2004 Subject: [HTTP-Recorder] HTTP::Recorder OK! Question about Javascript, though... Message-ID: <5C1F93C6ACBCBA47A037A38A2C214B74A623@dromail2.co.corp.verintsys tems.com> Hi... I read the documentation wherein it says "HTTP::Recorder won't record Javascript actions", but I wonder-- it seems like Javascript should produce *some* interaction with the server that would be visible to the proxy and thus to HTTP::Recorder. But apparently that is not the case. I've tested this several ways. I would really like to know why that is, if anyone on the list could supply an answer or a direction for study. -Chris From leira at mit.edu Mon Jun 14 21:57:05 2004 From: leira at mit.edu (Linda L. Julien) Date: Mon Jun 14 21:56:25 2004 Subject: [HTTP-Recorder] Sorry for my slow responses... Message-ID: <200406150157.VAA12785@small-gods.mit.edu> Hi folks, Thanks you all for using HTTP::Recorder. I discovered just a couple of days ago that I'd been dropped from the mailing list when the host's server had some problems, and then I promptly left town on a business trip. When I get back to a place with a real net connection (this bluetooth phone is great, but it sure is slow), I'll get started on answering all of your questions. In short, yes, 0.03_01 is the latest version, and is significantly better than 0.02. I recommend that you use that, and I'll bump the version number (so it's not listed as a developer release) as soon as I get a chance. Linda From aidan.kehoe at phorest.ie Fri Jun 11 11:57:18 2004 From: aidan.kehoe at phorest.ie (Aidan Kehoe) Date: Mon Jun 14 22:00:20 2004 Subject: [HTTP-Recorder] Another issue with forms and Message-ID: <40C9D65E.4040706@phorest.ie> Hi, When Recorder.pm piggybacks the various form details in the HTML that's sent to the browser, it puts tags immediately after several form elements. This breaks for as HTML, and rather displays it as text in the form box. I'm working around this with the attached patch--hopefully this will be useful for other users of the package. Bye, best regards, -- Aidan Kehoe Technical Officer nDevor Systems Ltd Email: aidan.kehoe@phorest.ie Mobile: +353 86 3874935 Phone: +353 1 8747800 (GMT office hours) Physical: 69 Middle Abbey Street, Dublin 1, Ireland Web: http://www.phorest.com/ -------------- next part -------------- --- Recorder.pm.orig Wed Jun 9 15:57:15 2004 +++ Recorder.pm Fri Jun 11 15:00:33 2004 @@ -269,7 +268,9 @@ push @forms, $token; $formnumber++; } - $newcontent .= ("<".$tagname); + + # Store the info for this tag before adding it to $newcontent. + my $taginfo = "<".$tagname; # keep the attributes in their original order my $attrlist = @$token[3]; @@ -283,13 +284,23 @@ $formfield = ("$prefix-form-". $formnumber."-".$attrs->{$attr}); } - $newcontent .= (" ".$attr."=\"".$attrs->{$attr}."\""); + $taginfo .= (" ".$attr."=\"".$attrs->{$attr}."\""); } - $newcontent .= (">\n"); + $taginfo .= (">\n"); + if ($formfield) { - $newcontent .= "\n"; + # In the event that this tag is a textarea, we can't put the + # hidden form value immediately after the opening, because + # it'll show up in the text field in the browser. So, we put + # it immediately before. + $taginfo = ''.$taginfo; $formfield = ""; } + + $newcontent .= $taginfo; + undef $taginfo; + if ($tagname eq 'form') { if (scalar @forms == 1) { $newcontent .= rewrite_form_content($attrs->{name}, From ivan at pechorin.com Mon Jun 14 22:32:59 2004 From: ivan at pechorin.com (Ivan Pechorin) Date: Mon Jun 14 22:32:30 2004 Subject: [HTTP-Recorder] Another issue with forms and In-Reply-To: <40C9D65E.4040706@phorest.ie> References: <40C9D65E.4040706@phorest.ie> Message-ID: <1087266779.17855.19.camel@localhost.localdomain> Hi! On Fri, 2004-06-11 at 19:57, Aidan Kehoe wrote: > When Recorder.pm piggybacks the various form details in the HTML that's sent to > the browser, it puts tags immediately after several > form elements. > > This breaks for as > HTML, and rather displays it as text in the form box. Aidan, could you try the latest version of HTTP::Recorder (0.03_01, marked as "developer release" on CPAN)? http://search.cpan.org/CPAN/authors/id/L/LE/LEIRA/HTTP-Recorder-0.03_01.tar.gz This issue was fixed in version 0.03_01. -- Ivan Pechorin From RGiersig at cpan.org Tue Jun 15 08:17:47 2004 From: RGiersig at cpan.org (Roland Giersig) Date: Tue Jun 15 08:17:09 2004 Subject: [HTTP-Recorder] HTTP::Recorder OK! Question about Javascript, though... In-Reply-To: <5C1F93C6ACBCBA47A037A38A2C214B74A623@dromail2.co.corp.verintsys tems.com> References: <5C1F93C6ACBCBA47A037A38A2C214B74A623@dromail2.co.corp.verintsys tems.com> Message-ID: <40CEE8EB.7060102@cpan.org> HTTP::Recorder will record all traffic between client and server, so even sites that make heavy use of JavaScript for user navigation should be recordable. The only thing that can cause problems is if a form is submitted via JavaScript instead of the
tag, as the JavaScript URL won't have the extra parameters attached. From what I see from the code, HTTP::Recorder does its best and logs it with GotoPage instead of FollowLink, so it isn't as fault-tolerant upon replay as the regular links. How did you test this? Note that a whole website (eg a webshop) could be implemented in only one JavaScript page, but the important thing for replay is: at some time the JavaScript has to send (POST) the data entered by the user to the server, and this will be recorded by HTTP::Recorder... Hope this helps, Roland McMahon, Chris wrote: > Hi... > I read the documentation wherein it says "HTTP::Recorder won't > record Javascript actions", but I wonder-- it seems like Javascript > should produce *some* interaction with the server that would be visible > to the proxy and thus to HTTP::Recorder. > But apparently that is not the case. I've tested this several > ways. I would really like to know why that is, if anyone on the list > could supply an answer or a direction for study. > -Chris > _______________________________________________ > HTTP-Recorder mailing list > HTTP-Recorder@lists.bestpractical.com > http://lists.bestpractical.com/cgi-bin/mailman/listinfo/http-recorder >