I have restored Gitlab from a backup, now every time I try to access the runners webpage, I get a Internal Server error.
I have tried uninstalling all configured gitlab-runners and accessing it, the problem persists. Here is a trace from the gitlab-rails/production.log:
Started GET "/admin/runners" for 127.0.0.1 at 2019-01-16 07:17:12 -0500
Processing by Admin::RunnersController#index as HTML
Completed 500 Internal Server Error in 66ms (ActiveRecord: 5.7ms)
ActionView::Template::Error ():
37:
38: .col-sm-6
39: .bs-callout
40: = render partial: 'ci/runner/how_to_setup_runner',
41: locals: { registration_token: Gitlab::CurrentSettings.runners_registration_token,
42: type: 'shared',
43: reset_token_url: reset_registration_token_admin_application_settings_path }
lib/gitlab/crypto_helper.rb:27:in `aes256_gcm_decrypt'
app/models/concerns/token_authenticatable_strategies/encrypted.rb:55:in `get_token'
app/models/concerns/token_authenticatable_strategies/base.rb:33:in `ensure_token!'
app/models/concerns/token_authenticatable.rb:43:in `block in add_authentication_token_field'
app/models/application_setting.rb:409:in `runners_registration_token'
lib/gitlab/current_settings.rb:19:in `method_missing'
app/views/admin/runners/index.html.haml:40:in `_app_views_admin_runners_index_html_haml___977288809002382166_69944849285200'
app/controllers/application_controller.rb:116:in `render'
lib/gitlab/i18n.rb:55:in `with_locale'
lib/gitlab/i18n.rb:61:in `with_user_locale'
app/controllers/application_controller.rb:420:in `set_locale'
lib/gitlab/middleware/multipart.rb:103:in `call'
lib/gitlab/request_profiler/middleware.rb:16:in `call'
lib/gitlab/middleware/go.rb:19:in `call'
lib/gitlab/etag_caching/middleware.rb:13:in `call'
lib/gitlab/middleware/correlation_id.rb:16:in `block in call'
lib/gitlab/correlation_id.rb:15:in `use_id'
lib/gitlab/middleware/correlation_id.rb:15:in `call'
lib/gitlab/middleware/read_only/controller.rb:40:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/request_context.rb:20:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:29:in `call'
lib/gitlab/middleware/release_env.rb:13:in `call'
Started GET "/-/metrics" for 127.0.0.1 at 2019-01-16 07:17:24 -0500
Processing by MetricsController#index as HTML
Completed 200 OK in 1ms (Views: 0.4ms | ActiveRecord: 0.0ms)
asked Jan 16, 2019 at 12:18
1
I could resolve the issue by following official gitlab documentation:
gitlab-rails dbconsole
-- Clear project tokens
UPDATE projects SET runners_token = null, runners_token_encrypted = null;
-- Clear group tokens
UPDATE namespaces SET runners_token = null, runners_token_encrypted = null;
-- Clear instance tokens
UPDATE application_settings SET runners_registration_token_encrypted = null;
-- Clear runner tokens
UPDATE ci_runners SET token = null, token_encrypted = null;
https://docs.gitlab.com/ee/raketasks/backup_restore.html#reset-runner-registration-tokens
Edit:
After clearing existing pipeline jobs (see above), I could still not open the ci settings page for some migrated projects where I had set environment variables. In this case try to remove them:
gitlab-rails dbconsole
SELECT * FROM ci_variables;
DELETE FROM ci_variables WHERE project_id='XX';
Hope that helps.
answered Oct 2, 2019 at 18:41
Tobias ErnstTobias Ernst
4,2141 gold badge32 silver badges30 bronze badges
1
You did a restore, but probably did not restore the secrets store (/etc/gitlab/gitlab-secrets.json or $GITLAB_HOME/config/secrets.json). Same issue as gitlab 500 errors in the admin area
You can reset it like this from the rails console on the gitlab server:
root@gitlab:/# gitlab-rails console
-------------------------------------------------------------------------------------
GitLab: 11.5.1 (c90ae59)
GitLab Shell: 8.4.1
postgresql: 9.6.8
-------------------------------------------------------------------------------------
Both Deployment and its :status machine have defined a different default for "status". Use only one or the other for defining defaults to avoid unexpected behaviors.
Loading production environment (Rails 4.2.10)
irb(main):001:0> ApplicationSetting.current.reset_runners_registration_token!
=> true
answered Jan 16, 2019 at 19:55
5
i’ve solved this,maybe it is usefal
a. Reset CI/CD variables
gitlab-rails dbconsole
SELECT * FROM public."ci_group_variables";
SELECT * FROM public."ci_variables";
DELETE FROM ci_group_variables;
DELETE FROM ci_variables;
b. Reset runner registration tokens
gitlab-rails dbconsole
-- Clear project tokens
UPDATE projects SET runners_token = null, runners_token_encrypted = null;
-- Clear group tokens
UPDATE namespaces SET runners_token = null, runners_token_encrypted = null;
-- Clear instance tokens
UPDATE application_settings SET runners_registration_token_encrypted = null;
UPDATE application_settings SET encrypted_ci_jwt_signing_key = null;
-- Clear runner tokens
UPDATE ci_runners SET token = null, token_encrypted = null;
c. Reset pending pipeline jobs
sudo gitlab-rails dbconsole
-- Clear build tokens
UPDATE ci_builds SET token = null, token_encrypted = null;
d. Fix project integrations
gitlab-rails dbconsole
-- truncate web_hooks table
TRUNCATE web_hooks CASCADE;
from gitlab
answered Mar 25, 2021 at 8:15
Loading
Here is what happen, i got either a 500 or 502 error after an upgrade using Omnibus method. And got the error message «Gitlab is not responding»
The first thing i did was to look for what could have happen. And the first place to see that is on the log file located at
/var/log/gitlab/gitlab-rails/production.log
And the log basically just give me an error as shown below,
Started GET "/users/sign_in" for 175.144.6.68 at 2015-12-12 23:48:54 +0800 Processing by SessionsController#new as HTML Completed 500 Internal Server Error in 98ms (ActiveRecord: 10.8ms) NoMethodError (undefined method `push_events=' for #<GitlabCiService:0x0000000463dba8>): app/models/project.rb:809:in `builds_enabled=' app/controllers/application_controller.rb:194:in `add_gon_variables'
But when i do a status check, it gives me this
[root@git gitlab-rails]# gitlab-ctl status run: gitlab-workhorse: (pid 4934) 1009s; run: log: (pid 4147) 1227s run: logrotate: (pid 4942) 1008s; run: log: (pid 296) 3434s run: nginx: (pid 4948) 1008s; run: log: (pid 299) 3434s run: postgresql: (pid 4957) 1007s; run: log: (pid 301) 3434s run: redis: (pid 4965) 1007s; run: log: (pid 294) 3434s run: sidekiq: (pid 4972) 1005s; run: log: (pid 302) 3434s run: unicorn: (pid 4990) 1004s; run: log: (pid 305) 3434s
I have pretty much no idea what is going on. But after trying out different ways, it seems to boil down to the following,
1. Check what is going on
Firing the following command should give you an idea what is going on with your configure.
sudo gitlab-rake gitlab:check
After that you could try see what is causing it.
2. Forget to turn on postgres before upgrade
Well, because gitlab said to shutdown gitlab before upgrading, hence i did this,
gitlab-ctl stop
which stops everything including postgres. Hence, database migration wasn’t possible. Therefore, i fire the following command and see whether that helps
gitlab-rake db:migrate
Now, after this i still got a 502 error but at least i’m not stuck with 500 error!
3. Forget to reconfigure after an upgrade
Well, if its not database migration, then every time you did a migration, remember to do a reconfigure!
gitlab-ctl reconfigure
Once i did this. Wait a while, and puff! The screen is back up!
I’m just grateful everything is ok! Just remember to back up your VM image before doing all these upgrades!
After adding several new files to my project, upon submitting a merge request Git presents an error with the following text;
«500. We’re sorry but something went wrong. Please contact your GitLab administrator if this problem persists.»
I have tested with a separate commit which worked fine suggesting that it is the content that I am pushing onto Git that is serving the error.
I have checked online such as the following threads —
https://github.com/gitlabhq/gitlabhq/issues/7183
https://github.com/gitlabhq/gitlabhq/issues/1774
Neither of which provide the solution.
The content I am adding includes some fonts, JS, css and an images folder — all of which work fine outside of Git.
I’m wondering if Git can pick up file names and produce errors if there are conflicts. Does anyone know if this is possible? Or any other ideas as to what the problem could be?
Thanks in advance.
Hello Everybody,
I recently set up an own gitlab ce instance on my homeserver. At first I had problems setting it up with my own nginx webserver. On my first install it was working but I could not clone over HTTP and when downloading a project as zip file, the zip was always empty
Then I removed the gitlab-ce package, threw away the /etc/gitlab directory and reinstalled the package. For this installation I followed these instructions: https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/doc/settings/nginx.md#using-a-non-bundled-web-server
I used the nginx config under https://gitlab.com/gitlab-org/gitlab-ce/blob/master/lib/support/nginx/gitlab-ssl and changed the upstream gitlab-workhorse part to omnibus and changed domains and certificate paths.
After this I now can clone and download zips correctly. However if I go to the admin settings and change something, after clicking the Save Changes button I always get a Error pache: 500 Internal Server Error. THis happens on all the settins in the admin area and as well when I select Overview->Runners in the navigation on the left. The settings I modified after the first installation are still active after my second installation.
Does anyone know what causes this behaviour and how I can fix it?
Thank you in advance and have a nice day!