Ошибка 110 connection timed out

I have Puma running as the upstream app server and Riak as my background db cluster. When I send a request that map-reduces a chunk of data for about 25K users and returns it from Riak to the app, I get an error in the Nginx log:

upstream timed out (110: Connection timed out) while reading
response header from upstream

If I query my upstream directly without nginx proxy, with the same request, I get the required data.

The Nginx timeout occurs once the proxy is put in.

**nginx.conf**

http {
    keepalive_timeout 10m;
    proxy_connect_timeout  600s;
    proxy_send_timeout  600s;
    proxy_read_timeout  600s;
    fastcgi_send_timeout 600s;
    fastcgi_read_timeout 600s;
    include /etc/nginx/sites-enabled/*.conf;
}

**virtual host conf**

upstream ss_api {
  server 127.0.0.1:3000 max_fails=0  fail_timeout=600;
}

server {
  listen 81;
  server_name xxxxx.com; # change to match your URL

  location / {
    # match the name of upstream directive which is defined above
    proxy_pass http://ss_api; 
    proxy_set_header  Host $http_host;
    proxy_set_header  X-Real-IP  $remote_addr;
    proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_cache cloud;
    proxy_cache_valid  200 302  60m;
    proxy_cache_valid  404      1m;
    proxy_cache_bypass $http_authorization;
    proxy_cache_bypass http://ss_api/account/;
    add_header X-Cache-Status $upstream_cache_status;
  }
}

Nginx has a bunch of timeout directives. I don’t know if I’m missing something important. Any help would be highly appreciated….

bschlueter's user avatar

bschlueter

3,8171 gold badge30 silver badges48 bronze badges

asked Sep 11, 2013 at 12:01

user2768537's user avatar

1

This happens because your upstream takes too long to answer the request and NGINX thinks the upstream already failed in processing the request, so it responds with an error.
Just include and increase proxy_read_timeout in location config block.
Same thing happened to me and I used 1 hour timeout for an internal app at work:

proxy_read_timeout 3600;

With this, NGINX will wait for an hour (3600s) for its upstream to return something.

Armen Michaeli's user avatar

answered Sep 13, 2017 at 19:51

Sergio Gonzalez's user avatar

Sergio GonzalezSergio Gonzalez

1,8611 gold badge12 silver badges12 bronze badges

3

You should always refrain from increasing the timeouts, I doubt your backend server response time is the issue here in any case.

I got around this issue by clearing the connection keep-alive flag and specifying http version as per the answer here:
https://stackoverflow.com/a/36589120/479632

server {
    location / {
        proxy_set_header   X-Real-IP $remote_addr;
        proxy_set_header   Host      $http_host;

        # these two lines here
        proxy_http_version 1.1;
        proxy_set_header Connection "";

        proxy_pass http://localhost:5000;
    }
}

Unfortunately I can’t explain why this works and didn’t manage to decipher it from the docs mentioned in the answer linked either so if anyone has an explanation I’d be very interested to hear it.

Community's user avatar

answered Apr 13, 2016 at 5:17

Almund's user avatar

AlmundAlmund

5,7353 gold badges31 silver badges35 bronze badges

13

First figure out which upstream is slowing by consulting the nginx error log
file and adjust the read time out accordingly
in my case it was fastCGI

2017/09/27 13:34:03 [error] 16559#16559: *14381 upstream timed out (110: Connection timed out) while reading response header from upstream, client:xxxxxxxxxxxxxxxxxxxxxxxxx", upstream: "fastcgi://unix:/var/run/php/php5.6-fpm.sock", host: "xxxxxxxxxxxxxxx", referrer: "xxxxxxxxxxxxxxxxxxxx"

So i have to adjust the fastcgi_read_timeout in my server configuration

 location ~ \.php$ {
     fastcgi_read_timeout 240;
     ...
 }

See: original post

Finwe's user avatar

Finwe

6,3812 gold badges29 silver badges44 bronze badges

answered Sep 27, 2017 at 14:19

Ruberandinda Patience's user avatar

2

In your case it helps a little optimization in proxy, or you can use «# time out settings»

location / 
{        

  # time out settings
  proxy_connect_timeout 159s;
  proxy_send_timeout   600;
  proxy_read_timeout   600;
  proxy_buffer_size    64k;
  proxy_buffers     16 32k;
  proxy_busy_buffers_size 64k;
  proxy_temp_file_write_size 64k;
  proxy_pass_header Set-Cookie;
  proxy_redirect     off;
  proxy_hide_header  Vary;
  proxy_set_header   Accept-Encoding '';
  proxy_ignore_headers Cache-Control Expires;
  proxy_set_header   Referer $http_referer;
  proxy_set_header   Host   $host;
  proxy_set_header   Cookie $http_cookie;
  proxy_set_header   X-Real-IP  $remote_addr;
  proxy_set_header X-Forwarded-Host $host;
  proxy_set_header X-Forwarded-Server $host;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}

answered Dec 19, 2013 at 10:13

Dimitrios's user avatar

DimitriosDimitrios

1,15311 silver badges10 bronze badges

4

I would recommend to look at the error_logs, specifically at the upstream part where it shows specific upstream that is timing out.

Then based on that you can adjust proxy_read_timeout, fastcgi_read_timeout or uwsgi_read_timeout.

Also make sure your config is loaded.

More details here Nginx upstream timed out (why and how to fix)

Eje's user avatar

Eje

3544 silver badges8 bronze badges

answered Apr 22, 2017 at 17:36

gansbrest's user avatar

gansbrestgansbrest

7891 gold badge8 silver badges11 bronze badges

1

I think this error can happen for various reasons, but it can be specific to the module you’re using. For example I saw this using the uwsgi module, so had to set «uwsgi_read_timeout».

answered Oct 10, 2013 at 10:50

Richard's user avatar

RichardRichard

1,81317 silver badges17 bronze badges

1

As many others have pointed out here, increasing the timeout settings for NGINX can solve your issue.

However, increasing your timeout settings might not be as straightforward as many of these answers suggest. I myself faced this issue and tried to change my timeout settings in the /etc/nginx/nginx.conf file, as almost everyone in these threads suggest. This did not help me a single bit; there was no apparent change in NGINX’ timeout settings. Now, many hours later, I finally managed to fix this problem.

The solution lies in this forum thread, and what it says is that you should put your timeout settings in /etc/nginx/conf.d/timeout.conf (and if this file doesn’t exist, you should create it). I used the same settings as suggested in the thread:

proxy_connect_timeout 600;
proxy_send_timeout 600;
proxy_read_timeout 600;
send_timeout 600;

answered Feb 9, 2019 at 9:54

Andreas Forslöw's user avatar

Please also check the keepalive_timeout of the upstream server.

I got a similar issue: random 502, with Connection reset by peer errors in nginx logs, happening when server was on heavy load. Eventually found it was caused by a mismatch between nginx’ and upstream’s (gunicorn in my case) keepalive_timeout values. Nginx was at 75s and upstream only a few seconds. This caused upstream to sometimes fall in timeout and drop the connection, while nginx didn’t understand why.

Raising the upstream server value to match nginx’ one solved the issue.

answered Jul 9, 2021 at 15:29

Eino Gourdin's user avatar

Eino GourdinEino Gourdin

4,1893 gold badges39 silver badges67 bronze badges

If you’re using an AWS EC2 instance running Linux like I am you may also need to restart Nginx for the changes to take effect after adding proxy_read_timeout 3600; to etc/nginx/nginx.conf, I did: sudo systemctl restart nginx

answered Jul 15, 2022 at 18:17

Amon's user avatar

AmonAmon

2,7355 gold badges30 silver badges53 bronze badges

I had the same problem and resulted that was an «every day» error in the rails controller. I don’t know why, but on production, puma runs the error again and again causing the message:

upstream timed out (110: Connection timed out) while reading response header from upstream

Probably because Nginx tries to get the data from puma again and again.The funny thing is that the error caused the timeout message even if I’m calling a different action in the controller, so, a single typo blocks all the app.

Check your log/puma.stderr.log file to see if that is the situation.

answered Dec 26, 2016 at 19:28

aarkerio's user avatar

aarkerioaarkerio

2,1932 gold badges20 silver badges34 bronze badges

Hopefully it helps someone:
I ran into this error and the cause was wrong permission on the log folder for phpfpm, after changing it so phpfpm could write to it, everything was fine.

answered Jan 3, 2019 at 1:08

Maurício Otta's user avatar

From our side it was using spdy with proxy cache. When the cache expires we get this error till the cache has been updated.

answered Jun 18, 2014 at 21:26

timhaak's user avatar

timhaaktimhaak

2,4032 gold badges14 silver badges7 bronze badges

For proxy_upstream timeout, I tried the above setting but these didn’t work.

Setting resolver_timeout worked for me, knowing it was taking 30s to produce the upstream timeout message. E.g. me.atwibble.com could not be resolved (110: Operation timed out).

http://nginx.org/en/docs/http/ngx_http_core_module.html#resolver_timeout

Eje's user avatar

Eje

3544 silver badges8 bronze badges

answered Nov 25, 2019 at 13:44

David Mercer's user avatar

we faced issue while saving content (customt content type) giving timeout error. Fixed this by adding all above timeouts, http client config to 600s and increasing memory for php process to 3gb.

answered Dec 10, 2021 at 5:44

Jagdish Bhadra's user avatar

1

If you are using wsl2 on windows 10, check your version by this command:

wsl -l -v

you should see 2 under the version.
if you don’t, you need to install wsl_update_x64.

Dijkgraaf's user avatar

Dijkgraaf

11.1k17 gold badges42 silver badges54 bronze badges

answered Jan 22, 2022 at 6:25

salman's user avatar

I test proxy_read_timeout 100s and find timeout 100s in access log,set 210s appeared,so you can set 600s or long with your web.
proxy_read_timeout
accesslog

answered Mar 10 at 5:17

Cooperd's user avatar

new add a line config to location or nginx.conf, for example:
proxy_read_timeout 900s;

answered Mar 19, 2021 at 10:57

leiting.liu's user avatar

1

I have Puma running as the upstream app server and Riak as my background db cluster. When I send a request that map-reduces a chunk of data for about 25K users and returns it from Riak to the app, I get an error in the Nginx log:

upstream timed out (110: Connection timed out) while reading
response header from upstream

If I query my upstream directly without nginx proxy, with the same request, I get the required data.

The Nginx timeout occurs once the proxy is put in.

**nginx.conf**

http {
    keepalive_timeout 10m;
    proxy_connect_timeout  600s;
    proxy_send_timeout  600s;
    proxy_read_timeout  600s;
    fastcgi_send_timeout 600s;
    fastcgi_read_timeout 600s;
    include /etc/nginx/sites-enabled/*.conf;
}

**virtual host conf**

upstream ss_api {
  server 127.0.0.1:3000 max_fails=0  fail_timeout=600;
}

server {
  listen 81;
  server_name xxxxx.com; # change to match your URL

  location / {
    # match the name of upstream directive which is defined above
    proxy_pass http://ss_api; 
    proxy_set_header  Host $http_host;
    proxy_set_header  X-Real-IP  $remote_addr;
    proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_cache cloud;
    proxy_cache_valid  200 302  60m;
    proxy_cache_valid  404      1m;
    proxy_cache_bypass $http_authorization;
    proxy_cache_bypass http://ss_api/account/;
    add_header X-Cache-Status $upstream_cache_status;
  }
}

Nginx has a bunch of timeout directives. I don’t know if I’m missing something important. Any help would be highly appreciated….

bschlueter's user avatar

bschlueter

3,8171 gold badge30 silver badges48 bronze badges

asked Sep 11, 2013 at 12:01

user2768537's user avatar

1

This happens because your upstream takes too long to answer the request and NGINX thinks the upstream already failed in processing the request, so it responds with an error.
Just include and increase proxy_read_timeout in location config block.
Same thing happened to me and I used 1 hour timeout for an internal app at work:

proxy_read_timeout 3600;

With this, NGINX will wait for an hour (3600s) for its upstream to return something.

Armen Michaeli's user avatar

answered Sep 13, 2017 at 19:51

Sergio Gonzalez's user avatar

Sergio GonzalezSergio Gonzalez

1,8611 gold badge12 silver badges12 bronze badges

3

You should always refrain from increasing the timeouts, I doubt your backend server response time is the issue here in any case.

I got around this issue by clearing the connection keep-alive flag and specifying http version as per the answer here:
https://stackoverflow.com/a/36589120/479632

server {
    location / {
        proxy_set_header   X-Real-IP $remote_addr;
        proxy_set_header   Host      $http_host;

        # these two lines here
        proxy_http_version 1.1;
        proxy_set_header Connection "";

        proxy_pass http://localhost:5000;
    }
}

Unfortunately I can’t explain why this works and didn’t manage to decipher it from the docs mentioned in the answer linked either so if anyone has an explanation I’d be very interested to hear it.

Community's user avatar

answered Apr 13, 2016 at 5:17

Almund's user avatar

AlmundAlmund

5,7353 gold badges31 silver badges35 bronze badges

13

First figure out which upstream is slowing by consulting the nginx error log
file and adjust the read time out accordingly
in my case it was fastCGI

2017/09/27 13:34:03 [error] 16559#16559: *14381 upstream timed out (110: Connection timed out) while reading response header from upstream, client:xxxxxxxxxxxxxxxxxxxxxxxxx", upstream: "fastcgi://unix:/var/run/php/php5.6-fpm.sock", host: "xxxxxxxxxxxxxxx", referrer: "xxxxxxxxxxxxxxxxxxxx"

So i have to adjust the fastcgi_read_timeout in my server configuration

 location ~ \.php$ {
     fastcgi_read_timeout 240;
     ...
 }

See: original post

Finwe's user avatar

Finwe

6,3812 gold badges29 silver badges44 bronze badges

answered Sep 27, 2017 at 14:19

Ruberandinda Patience's user avatar

2

In your case it helps a little optimization in proxy, or you can use «# time out settings»

location / 
{        

  # time out settings
  proxy_connect_timeout 159s;
  proxy_send_timeout   600;
  proxy_read_timeout   600;
  proxy_buffer_size    64k;
  proxy_buffers     16 32k;
  proxy_busy_buffers_size 64k;
  proxy_temp_file_write_size 64k;
  proxy_pass_header Set-Cookie;
  proxy_redirect     off;
  proxy_hide_header  Vary;
  proxy_set_header   Accept-Encoding '';
  proxy_ignore_headers Cache-Control Expires;
  proxy_set_header   Referer $http_referer;
  proxy_set_header   Host   $host;
  proxy_set_header   Cookie $http_cookie;
  proxy_set_header   X-Real-IP  $remote_addr;
  proxy_set_header X-Forwarded-Host $host;
  proxy_set_header X-Forwarded-Server $host;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}

answered Dec 19, 2013 at 10:13

Dimitrios's user avatar

DimitriosDimitrios

1,15311 silver badges10 bronze badges

4

I would recommend to look at the error_logs, specifically at the upstream part where it shows specific upstream that is timing out.

Then based on that you can adjust proxy_read_timeout, fastcgi_read_timeout or uwsgi_read_timeout.

Also make sure your config is loaded.

More details here Nginx upstream timed out (why and how to fix)

Eje's user avatar

Eje

3544 silver badges8 bronze badges

answered Apr 22, 2017 at 17:36

gansbrest's user avatar

gansbrestgansbrest

7891 gold badge8 silver badges11 bronze badges

1

I think this error can happen for various reasons, but it can be specific to the module you’re using. For example I saw this using the uwsgi module, so had to set «uwsgi_read_timeout».

answered Oct 10, 2013 at 10:50

Richard's user avatar

RichardRichard

1,81317 silver badges17 bronze badges

1

As many others have pointed out here, increasing the timeout settings for NGINX can solve your issue.

However, increasing your timeout settings might not be as straightforward as many of these answers suggest. I myself faced this issue and tried to change my timeout settings in the /etc/nginx/nginx.conf file, as almost everyone in these threads suggest. This did not help me a single bit; there was no apparent change in NGINX’ timeout settings. Now, many hours later, I finally managed to fix this problem.

The solution lies in this forum thread, and what it says is that you should put your timeout settings in /etc/nginx/conf.d/timeout.conf (and if this file doesn’t exist, you should create it). I used the same settings as suggested in the thread:

proxy_connect_timeout 600;
proxy_send_timeout 600;
proxy_read_timeout 600;
send_timeout 600;

answered Feb 9, 2019 at 9:54

Andreas Forslöw's user avatar

Please also check the keepalive_timeout of the upstream server.

I got a similar issue: random 502, with Connection reset by peer errors in nginx logs, happening when server was on heavy load. Eventually found it was caused by a mismatch between nginx’ and upstream’s (gunicorn in my case) keepalive_timeout values. Nginx was at 75s and upstream only a few seconds. This caused upstream to sometimes fall in timeout and drop the connection, while nginx didn’t understand why.

Raising the upstream server value to match nginx’ one solved the issue.

answered Jul 9, 2021 at 15:29

Eino Gourdin's user avatar

Eino GourdinEino Gourdin

4,1893 gold badges39 silver badges67 bronze badges

If you’re using an AWS EC2 instance running Linux like I am you may also need to restart Nginx for the changes to take effect after adding proxy_read_timeout 3600; to etc/nginx/nginx.conf, I did: sudo systemctl restart nginx

answered Jul 15, 2022 at 18:17

Amon's user avatar

AmonAmon

2,7355 gold badges30 silver badges53 bronze badges

I had the same problem and resulted that was an «every day» error in the rails controller. I don’t know why, but on production, puma runs the error again and again causing the message:

upstream timed out (110: Connection timed out) while reading response header from upstream

Probably because Nginx tries to get the data from puma again and again.The funny thing is that the error caused the timeout message even if I’m calling a different action in the controller, so, a single typo blocks all the app.

Check your log/puma.stderr.log file to see if that is the situation.

answered Dec 26, 2016 at 19:28

aarkerio's user avatar

aarkerioaarkerio

2,1932 gold badges20 silver badges34 bronze badges

Hopefully it helps someone:
I ran into this error and the cause was wrong permission on the log folder for phpfpm, after changing it so phpfpm could write to it, everything was fine.

answered Jan 3, 2019 at 1:08

Maurício Otta's user avatar

From our side it was using spdy with proxy cache. When the cache expires we get this error till the cache has been updated.

answered Jun 18, 2014 at 21:26

timhaak's user avatar

timhaaktimhaak

2,4032 gold badges14 silver badges7 bronze badges

For proxy_upstream timeout, I tried the above setting but these didn’t work.

Setting resolver_timeout worked for me, knowing it was taking 30s to produce the upstream timeout message. E.g. me.atwibble.com could not be resolved (110: Operation timed out).

http://nginx.org/en/docs/http/ngx_http_core_module.html#resolver_timeout

Eje's user avatar

Eje

3544 silver badges8 bronze badges

answered Nov 25, 2019 at 13:44

David Mercer's user avatar

we faced issue while saving content (customt content type) giving timeout error. Fixed this by adding all above timeouts, http client config to 600s and increasing memory for php process to 3gb.

answered Dec 10, 2021 at 5:44

Jagdish Bhadra's user avatar

1

If you are using wsl2 on windows 10, check your version by this command:

wsl -l -v

you should see 2 under the version.
if you don’t, you need to install wsl_update_x64.

Dijkgraaf's user avatar

Dijkgraaf

11.1k17 gold badges42 silver badges54 bronze badges

answered Jan 22, 2022 at 6:25

salman's user avatar

I test proxy_read_timeout 100s and find timeout 100s in access log,set 210s appeared,so you can set 600s or long with your web.
proxy_read_timeout
accesslog

answered Mar 10 at 5:17

Cooperd's user avatar

new add a line config to location or nginx.conf, for example:
proxy_read_timeout 900s;

answered Mar 19, 2021 at 10:57

leiting.liu's user avatar

1

Исправление ошибки “Upstream timed out (110: Connection timed out)”, из-за долгого ответа сервера.

[error] upstream timed out (110: Connection timed out) while reading response header from upstream, 
client: xxx.xxx.xxx.xxx, server: mydomain.com, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8080", host: "mydomain.com", referrer: "requested_url"

Вносим изменения в конфигурационный файл nginx.conf в директиву proxy_read_timeout, а именно — увеличим значение при котором сервер будет ожидать ответа на запрос. По умолчанию 60 секунд, мы выставим — 300 секунд.

server {
listen 80;
server_name mydomain.com;

location / {
...
proxy_read_timeout 300;
...
}
...
}

Это решит возникшую проблему.

Thank you for reading this post, don’t forget to subscribe!

ошиб­ка upstream timed out (110: Connection timed out) может воз­ни­кать в двух слу­ча­ях. При­чем назва­ние самой ошиб­ки ука­зы­ва­ет на реше­ние — необ­хо­ди­мо уве­ли­чить вре­мя ожи­да­ния в настрой­ках веб-сервера.

Nginx в качестве proxy или reverse proxy

В этом слу­чае ошиб­ка может воз­ни­кать, если истек­ло вре­мя ожи­да­ния на чте­ние отве­та от прокси-сервера.

То есть Nginx отпра­вил запрос и не дождал­ся отве­та. Если вы уве­ре­ны, что ваше веб-при­ло­же­ние рабо­та­ет кор­рект­но, то необ­хо­ди­мо уве­ли­чить этот тай­маут в фай­ле кон­фи­гу­ра­ции nginx.conf в сек­ции locatio

[codesyntax lang=»php»]

location / {

...

proxy_send_timeout 150;

proxy_read_timeout 150;

...

}

[/codesyntax]

Уста­нов­ка вре­ме­ни ожи­да­ния на отправ­ку и чте­ние отве­та, в секундах

Nginx с подключенными FastCGI-серверами

В этом слу­чае ошиб­ка воз­ни­ка­ет, если истек­ло вре­мя ожи­да­ния на чте­ние отве­та от под­клю­чен­ных сер­ви­сов или при­ло­же­ний, PHP-FPM, к примеру.

Реше­ние такое же баналь­ное, как и в пер­вом слу­чае — необ­хо­ди­мо уве­ли­чить вре­мя ожидания:

[codesyntax lang=»php»]

location ~* .php$ {

include   fastcgi_params;

...

fastcgi_read_timeout 150;

...

}

[/codesyntax]

Самое главное

Преж­де чем уве­ли­чить вре­мя ожи­да­ния, кото­рое в дан­ном слу­чае по умол­ча­нию состав­ля­ет 60 с, сле­ду­ет про­ве­рить рабо­то­спо­соб­ность всех ком­по­нен­тов и моду­лей. Если же все рабо­та­ет как нуж­но, то уве­ли­че­ние тай­мау­та будет самым про­стым реше­ни­ем проблемы.

https://github.com/midnight47/

Периодически появляются ошибки «upstream timed out (110: Connection timed out) while connecting to upstream», особенно когда какой нибудь бот индексирует страницы. Понятно, что это означает, что сервер вовремя не вернул результат. Но сами страницы отдаются очень быстро, на них нету slow queries, да и сервер особо не нагружен. Что ещё может вызывать такие ошибки? Куда можно копать?


  • Вопрос задан

  • 6346 просмотров

Пригласить эксперта

Варианты:
1) на бекенде к которому обращается nginx кончаются коннекты
2) На сервере кончаются дескрипторы

Как обычно информации мало, в проблеме так не разобраться.

Как правило ошибка лечится увеличением
proxy_read_timeout 300s;
еще посмотрите вот эти директивы:
proxy_connect_timeout 300s;
proxy_send_timeout 300s;
Они все по дефолту 60 сек
+ если у вас php-fpm, могут быть
fastcgi_send_timeout
fastcgi_read_timeout
Смотрите всё, тестируйте, проверяйте


  • Показать ещё
    Загружается…

22 сент. 2023, в 10:57

5000 руб./за проект

22 сент. 2023, в 10:55

4000 руб./за проект

22 сент. 2023, в 10:55

5000 руб./за проект

Минуточку внимания

Понравилась статья? Поделить с друзьями:
  • Ошибка 11 ринай котел риннай
  • Ошибка 11 при установке спор
  • Ошибка 111 xiaomi
  • Ошибка 111 gta 2
  • Ошибка 1109 опель