Ошибка 503 curl

• The RFC states that proxies may return error content (just as non-proxy error responses may contain content)
• curl returns non-proxy error content presumably without differentiating it from non-error content
• curl returns proxy error headers (via -I)
• proxy error headers refer to the proxy error content and are in the same transport channel as that content
• curl squelches the proxy error content, even while processing the proxy error headers
• curl is frequently used BOTH by end-users fetching resource AND developers testing HTTP infrastructure

I don’t see how the RFC is relevant

I … uh, disagree.

curl does transfers to/from a given URL. The proxy is a middle man

There’s a lot to unpack there. I know you’re speaking loosely but: URLs identify resources and the means by which to access those. The transfers are not to/from a given URL but rather to/from the myriad of services involved in providing the resource in accordance with the specified protocol.

Specifically for HTTP(S) there are some inherent ambiguities in the protocol because there is exactly one response channel (the tcp connection on which the request is made), but the returned data may or may not be returned by the referenced resource due to transports (and errors) reaching it. The content returned may or may not be the URL’s content per the spec (@jay’s comment.)

Due to this, no one SHOULD assume the content returned by HTTP during proxying MUST be the destination’s content.

How do you propose curl delivers that to make sure that users don’t mistake it for actual content?

I added the emphasis because clearly this is about «usability» concerns governing curl’s behavior/implementation relative to the protocols. A related question would be «how does curl make sure that users don’t mistake a non-proxy error response for actual content? Per @dfandrich’s comment it sounds like it makes no such effort.

Therefore, I would argue that proxy error content should not be demoted; particularly since curl must be configured to proxy, via command lines and/or env variables. Presumably then, the user is keenly aware that there is an additional error actor in the pipeline.

Given the fact that non-proxy error content is returned by curl in its current implementation, perhaps the better question is: How do you propose curl delivers all errors (not just those generated by the proxy) to ensure that users know the response is an error vs the «actual content»?

If there is a mechanism to do that, we can then ask the secondary question: «How do we differentiate an error response that is returned by the proxy vs one returned by the destination?»

So let me instead focus on the first part of the original question:

How do you propose curl delivers [the proxy error content]

Due to the foregoing, IMHO it would be neither unreasonable nor incorrect for curl to pass that content back exactly as the spec allows, inline in the channel, as «the response» which is an error response. This is also consistent with curl returning the body of error responses when the destination returns an error not via a proxy.

It seems that part of the concern here may be related to the difference between transparent proxies and non-transparent proxies. A goal of transparent proxying is to ensure the proxy is an unknown/unseen actor. I don’t think that’s the goal when configuring a proxy for curl. The proxy is explicitly known, and its operation is part of the pipeline. This observation may point to implementation choices/compromises to deal with all this. E.g., a «transparent proxy» option would cause curl to do the kinds of things being discussed here, that hide the existence of a proxy (such as eating proxy errors).

I’ll again note that curl’s current behavior is to report proxy errors via the -I option consistent with the error response being «the response» (vs differentiating those headers as «proxy headers»). Despite this «-I» behavior curl is then inconsistent in that it squelches the content to which those headers refer e.g. in the Content-Length etc.

Given the RFC, I would argue that curl (and libcurl) should probably deliver the content the same way it delivers non-proxy error content, because that’s what the RFC says is allowed by the protocol. If possible, I would have curl mark error content (whether proxy or non-proxy) in some manner consistent with its current practices, possibly with the addition of some kind of signaling to differentiate destination vs proxy errors. If it’s a libcurl issue, perhaps additional return values, bits in those values, struct pointers to be filled in with flags, etc can be used for that purpose (I’m not conversant with the library APIs).

data that isn’t the response body and isn’t really «headers» either

That this data «isn’t the [response from the ultimate destination] body and isn’t really ‘headers’ either» does not seem terribly relevant here. It is in fact the HTTP response content, and curl should return the HTTP response content, just as it returns the HTTP error response content when not proxied.

By the way, what does curl currently do about proxy errors for non-CONNECT proxies? My guess is those (and the associated content body) are returned «normally.»

Arguably, it’s the responsibility of the user of curl to differentiate between errors and actual content. Arguably, the user is in an excellent position to do so, given that they configured the proxy (e.g., by supplying -x). If curl can provide additional signaling (an inserted header? stderr output? highlighting? dunno) to indicate the error originated while attempting the CONNECT or pursuing the PROXY, that’s great!

Finally, if curl disagrees with my opinions or feels that it needs to protect users from «proxy generated content», then I’d ask you to consider not completely squelching that content always. An additional command-line option could be provided to let it through or to write it to a specified file (similar to -w).

If the core issue is some aspect of libcurl’s API I can’t speak to that, but on general principles, it would seem very possible to provide access to that content via the same mechanism that returns the destination content; and could acceptably do so only when signaling the special content type via API parameters, bitfields in the return values, etc etec or when the facility is requested via API parameters.

Thanks for listening. I’ll leave the conclusion in your capable hands.

I have some code which downloads a Excel document from a website, the document is quite large, however other documents which are larger seem to be working ok. Just seems to be this one that is returning 503 errors when I try I download it.

<?php
    $source = "https://mydomain.service-now.com/myTableName.do?EXCEL";
    $SN['username'] = "myUsername";
    $SN['password'] = "myPassword";

    $fp = fopen("test_myTableName.xls", 'w');
    $ch = curl_init($source);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
    curl_setopt($ch, CURLOPT_HTTPAUTH, CURLAUTH_ANY ) ;
    curl_setopt($ch, CURLOPT_USERPWD, $SN['username'] . ":" . $SN['password']);
    curl_setopt($ch, CURLOPT_FILE, $fp);
    curl_setopt($ch, CURLOPT_HEADER, false);
    curl_setopt($ch, CURLOPT_VERBOSE, 1);
    curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
    curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
    curl_setopt($ch, CURLOPT_CONNECTTIMEOUT ,0); 
    curl_setopt($ch, CURLOPT_TIMEOUT, 600);

    curl_setopt($ch, CURLOPT_FAILONERROR, 0);

    try {
      curl_exec($ch);
      echo "Successfully downloaded " . $tableID . " without error";
    } catch (Exception $e) {
      echo "Finished downloading " . $tableID . " with error - " . curl_error($ch);      
      exit();
    }
?>

Open in new window

What I fail to understand is my catch is never run, and I get nothing in my error_logs.

If I download the file using my browser it all works fine, just need it to authenticate with credentials, hence having to use curl.

Does anyone have any suggestions? Or anyway of looking further into the logs for what the error is?

Thank you

1. Home

2. php — cURL, script returning 503 Error (service unavailable)

933 votes

1 answers

Get the solution ↓↓↓

i am trying to login on the page ‘http://portal.demo.ascio.com/Logon.aspx’ with cURL but i am getting this error «503 Service Unavailable
No server is available to handle this request. «.
I am sending right POST which i get from firebug by login to the normal page.
I have been searching for a while and figured out that maybe i am not sending right header.

Code is:

$url = 'http://portal.demo.ascio.com/Logon.aspx';

$login_string = 'THERE ARE MY POSTS WHICH CONTAINS PASSWORD...thats is definitelly right'
$headers = array();                                                                                                            
$headers[] = "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
$headers[] = "Connection: Keep-Alive";                                         
$headers[] = "Host: portal.demo.ascio.com";                                    
$headers[] = "Referer: http://portal.demo.ascio.com/Logon.aspx";               
$headers[] = "Accept-Encoding: gzip, deflate";                                 
$headers[] = "Accept-Language: en-US,en;q=0.5";                                
$headers[] = "Content-type: application/x-www-form-urlencoded";                

//open connection                                                                                                              
$ch = curl_init();                                                                                                             

//set the url, number of POST vars, POST data                                  
curl_setopt($ch, CURLOPT_HEADER, true);                                        
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);                                
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);                                
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);                               
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);                               
curl_setopt($ch, CURLOPT_USERAGENT,'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:30.0) Gecko/20100101 Firefox/30.0');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, false);                               
curl_setopt($ch, CURLOPT_COOKIEJAR, 'cookie.txt');                             
curl_setopt($ch, CURLOPT_COOKIEFILE, 'cookie.txt');                            
curl_setopt($ch, CURLOPT_URL, $url);                                           
curl_setopt($ch, CURLOPT_POST, 5);                                              
curl_setopt($ch, CURLOPT_POSTFIELDS, $login_string);                             

$result = curl_exec($ch);                                                                                                      

print $result;                                                                                                                                                                                                                                     

curl_close($ch);

Thx for help

2022-04-4

Write your answer

---

314

votes

Answer

Solution:

In general, if you get the error of 500 series, such as 503, it means your request made it to the server side, and something in your input has caused the server to «break», maybe by design of the server itself. Double-check what you are sending and maybe also compare against the service API/Specs.

---

Share solution ↓

Additional Information:

Date the issue was resolved:

2022-04-4

Link To Source

Link To Answer
People are also looking for solutions of the problem: the browser (or proxy) sent a request that this server could not understand.

Didn’t find the answer?

Our community is visited by hundreds of web development professionals every day. Ask your question and get a quick answer for free.

---

Similar questions

Find the answer in similar questions on our website.

Write quick answer

Do you know the answer to this question? Write a quick response to it. With your help, we will make our community stronger.

About the technologies asked in this question

PHP

PHP (from the English Hypertext Preprocessor — hypertext preprocessor) is a scripting programming language for developing web applications. Supported by most hosting providers, it is one of the most popular tools for creating dynamic websites.
The PHP scripting language has gained wide popularity due to its processing speed, simplicity, cross-platform, functionality and distribution of source codes under its own license.
https://www.php.net/

HTML

HTML (English «hyper text markup language» — hypertext markup language) is a special markup language that is used to create sites on the Internet.
Browsers understand html perfectly and can interpret it in an understandable way. In general, any page on the site is html-code, which the browser translates into a user-friendly form. By the way, the code of any page is available to everyone.
https://www.w3.org/html/

Сервер при частых запросах отдает 503-й код.
Если повторить через 3 секунды — можно снова парсить.
Этот код выдает ошибку (текст ошибки под кодом):

function get_url($href) {
	    $curl = curl_init();
	    curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, FALSE);
	    curl_setopt($curl, CURLOPT_HEADER, false);
	    curl_setopt($curl, CURLOPT_FOLLOWLOCATION, true);
	    curl_setopt($curl, CURLOPT_URL, $href);
	    curl_setopt($curl, CURLOPT_REFERER, $href);
	    curl_setopt($curl, CURLOPT_RETURNTRANSFER, TRUE);
	    curl_setopt($curl, CURLOPT_USERAGENT, "my_useragent_string");
	    $str = curl_exec($curl);
	    $httpcode = curl_getinfo($curl, CURLINFO_HTTP_CODE);
	    if ($httpcode == '503') {
	    	curl_close($curl);
	    	sleep(4);
	    	$this->get_url($href);
	    }
	    curl_close($curl);
            return $str	
	}

Текст ошибки:
Message: curl_close(): 38 is not a valid cURL handle resource
Как грамотно оформить такую реализацию-обход частых запросов?
(смена useragent не помогает)